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PREFACE 


This  paper  was  prepared  by  the  Institute  for  Defense  Analyses 
(IDA)  for  the  Office  of  the  Assistant  Secretary  of  Defense  (OASD), 
Manpower,  Reserve  Affairs  and  Logistics  (MRA&L),  under  Contract 
MDA903  79  C  0320,  Task  Order  8l-l. 

The  purpose  of  the  paper  (and  the  workshop  which  it  reports 
on)  was  threefold:  to  assess  the  state  of  the  art  in  built-in¬ 
test  (BIT)  equipment  with  particular  emphasis  on  specification, 
testing  and  evaluation;  to  develop  guidelines  for  specifying 
requirements  for  BIT  and  for  its  test  and  evaluation;  and  to 
identify  areas  for  research  in  BIT  specification,  development  and 
mechanization . 

The  task  commenced  on  1  October  1980  and  concluded  on  28 
February  1981;  draft  publication  of  the  workshop  results  was 
scheduled  for  28  February  1981,  and  final  publication  for  30 
April  1931.  This  publication  is  issued  in  fulfillment  of  the 
contract . 
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EXECUTIVE  SUMMARY 


The  Office  of  the  Secretary  of  Defense  (MRA&L)  sponsored 
a  workshop  to  assess  progress  and  problems  in  specifying  and 
testing  Built-In-Test  (BIT)  used  in  complex  electronic  equip¬ 
ment.  The  workshop's  principal  recommendation  is  that  the 
current  specification  and  test  approach  be  broadened  to  include 
all  capabilities  associated  with  the  detection  and  isolation  of 
faults.  Current  practices  generally  address  only  a  narrow  sub¬ 
set  of  these  capabilities,  that  is,  BIT.  The  workshop  parti¬ 
cipants  defined  this  broad  capability  as  "100  Percent  Diag¬ 
nostics."1  The  diagnostic  capability  is  considered  to  have 
two  components — "automatic"  and  "manual."  The  automatic 
component  consists  of  BIT  or  semi-automatic  BIT  with  technical 
manuals,  while  the  rranual  component  consists  of  personnel  using 
logic,  external  test  equipment  and/or  manual  test  procedures. 
Observations  on  current  experience  with  BIT,  recommendations  to 
improve  specification  and  testing  of  "100  Percent  Diagnostics" 
that  can  be  put  into  practice  in  the  near  term,  and  proposed 
research  areas  are  discussed  below. 

The  workshop,  with  both  industry  and  the  Services  repre¬ 
sented,  was  organized  around  a  case  study/discussion  format. 
Presentations  were  selected  to  illustrate  successful  examples 
of  BIT  specification,  test  and  evaluation  and  to  represent  a 
wide  range  of  applications.  These  case  studies  were  presented 
to  individuals  with  experience  in  specification,  design,  test 


‘This  terminology  was  used  by  workshop  participants  and  will  be  used 
throughout  the  workshop  proceedings.  There  is  no  accepted  "standard" 
terminology  in  this  area. 


and  evaluation  of  BIT.  Workshop  recommendations  were  presented 
on  the  final  day  to  senior  Service  and  Office  of  the  Secretary 
of  Defense  managers. 


A.  OBSERVATIONS 

The  following  summarises  those  observations  on  the  perform¬ 
ance  of  systems  with  BIT  which  were  consistently  made  by  the 
various  participants  during  workshop  discussions. 

•  BIT-equipped  weapon  system  electronic  subsystems  (and 
equipment)  being  introduced  into  the  field  today  are 
not  meeting  the  diagnostic  specifications  which  are 
generally  in  the  range  of  90  to  95  percent  probability 
of  automatic  (or  semi-automatic)  fault  defection  and 
isolation.  In  addition*  experience  shows  that  20  to 
40  percent  of  the  items  which  were  replaced  because  of 
a  failure  indication  by  BIT  are  later  found  to  have  no 
failure  (principally  based  on  data  from  both  military 
and  civilian  aircraft  maintenance  experience).  The 
unnecessary  removal  rates  are  not  often  part  of 

this  specification. 

•  Even  where  BIT  specifications  are  close  to  being  met, 
the  systems  have  been  found  difficult  to  maintain,  as 
indicated  by  long  troubleshooting  times. 

•  BIT,  in  general,  is  not  designed  to  detect  all  failure 
conditions  (such  as  out-of-tolerance  conditions  or 
simultaneous  failures).  Consequently,  manual  trouble¬ 
shooting  is  required  to  augment  the  automatic  (BIT) 
capability  and  is  particularly  needed  for  the  more 
difficult  maintenance  problems. 

•  Manpower  planning  based  on  the  use  of  low  skilled 
technicians  (i.e.,  not  trained  in  system  operation) 
for  system  maintenance  combined  with  BIT  capability 
frequently  has  to  be  changed.  High  skilled  techni¬ 
cians  who  understand  system  operations  in  order  to 
correct  those  discrepancies  not  solved  by  low  skill 
level  plus  BIT  combination  have  had  to  be  included. 

•  Today's  state  of  the  art  for  mechanization  of  BIT 
capability  is  not  advanced  to  the  point  where  the 
requirement  for  highly  skilled  technicians  familiar 
with  troubleshooting  can  be  eliminated. 

•  There  is  little  visibility  to  program  management  dur¬ 
ing  system  development  of  the  progress  of  the  BIT 
design. 
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•  Adequate  test  time  (or  test  articles)  generally  has 
not  been  set  aside  to  develop  BIT  in  complex  systems 
prior  to  putting  these  systems  in  the  field. 

»  Contractual  BIT  laboratory  demonstration  tests  (MIL- 
STD-^Tl)  do  not  provide  reliable  predictions  of  BIT 
performance  in  the  field. 

•  BIT  contractual  specification  requirements  are  open  to 
a  wide  range  of  interpretations. 

•  Early  assessment  (initial  operations  test  and  evalua¬ 
tion)  of  field  operational  BIT  performance  is  very 
difficult  because  of  incomplete  software  and  because  ' 
of  the  interactions  between  operational  and  mainten¬ 
ance  personnel,  test  equipment  and  technical  manuals. 
Also,  standard  Service  maintenance  data  reporting 
systems  do  not  provide  sufficient  information  to 
evaluate  BIT  performance  or  to  solve  BIT  associated 
problems . 

•  About  two  years  of  field  operations,  using  dedicated 
technical  personnel  and  closed-loop  data  systems,  has 
been  found  necessary  to  mature  the  BIT  in  complex 
systems.  (Contractor  participation  has  been  required 
during  this  period. ) 


B.  RECOMMENDATIONS 

The  consensus  recommendations  developed  by  the  working 
groups  for  specifying  and  testing  diagnostics,  including  BIT 
are  presented  below  in  the  following  general  order:  develop¬ 
ment  of  performance  specifications,  contract  requirements,  and 
testing  and  evaluation. 

(1)  SPECIFICATIONS  FOR  DIAGNOSTICS  SHOULD  BE  DEVELOPED 

TO  MEET  USER-DEFINED  CONSTRAINTS.  The  equipment  user  needs  to 
identify  constraints  in  the  form  of  operational  and  maintenance 
parameters  such  as  turnaround  time,  maximum  down  time,  man¬ 
power  levels,  skill  levels,  and  self-sufficiency  in  deployment. 

(2)  THE  USER  OR  PROCURING  ACTIVITY  SHOULD  NOT  ARBITRARILY 
SPECIFY  A  LEVEL  OF  BIT  PERFORMANCE.  BIT  specifications  should 
evolve  from  consideration  of  other  diagnostic  specification 
requirements,  design,  manpower  and  support  constraints  as  well 
as  an  assessment  of  the  state  of  the  art  as  discussed  in  the 
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following  recommendation.  The  BIT  specifications  should  repre¬ 
sent  the  "best"  combination  of  automatic  and  manual  performance 
to  meet  the  user-defined  constraints  with  available  technology. 

(3)  CONTRACT  SPECIFICATIONS  FOR  DIAGNOSTICS  MUST  BE  IN¬ 
CLUDED  IN  THE  INITIAL  DEVELOPMENT  CONTRACT  SO  AS  TO  BE  AN 
INTEGRAL  PART  OF  EARLY  DESIGN  EFFORTS.  Contract  specifications 
for  diagnostics  should  evolve  from  both  user-defined  con¬ 
straints  and  the  design  process.  However,  based  on  observa¬ 
tions  of  the  performance  of  current  diagnostic  systems,  it  is 
apparent  that  many  of  the  contract  specifications  establish 
unrealistic  performance  levels  (for  example,  BIT  percentage  of 
faults  detected  and  isolated).  But,  performance  levels  for 
diagnostic  performance  need  not  be  completely  specified  in  the 
initial  development  contract.  Performance  levels  for  some  of 
the  diagnostic  terms  (defined  in  recommendations  6  and  7)  will 
be  directly  derivable  from  user-defined  constraints  and  know¬ 
ledge  of  existing  hardward  and  maintenance  capability.  For 
other  diagnostic  terms,  realistic,  achievable  levels  of  per¬ 
formance  may  not  be  readily  visible  at  the  start  of  the  design 
process.  In  this  latter  case,  diagnostic  performance  levels 
will  have  to  evolve  through  the  design  process  and  associated 
comparability  studies.  Such  factors  as  current  levels  of  BIT 
performance,  current  and  projected  skills  of  personnel  respon¬ 
sible  for  maintenance  and  technology  capabilities  have  to  be 
considered  in  establishing  realistic  achievable  performance 
levels  for  the  diagnostics.  There  was  no  general  consensus 
on  the  exact  process  for  establishing  these  performance  levels 
(this  area  will  need  additional  research).  However,  it  was 
agreed  that  all  performance  requirements  should  be  firmly 
established  as  contractual  requirements  before  significant 
system  design  efforts  are  initiated,  generally  in  the  full 
scale  development  contract. 
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(4)  SERVICES  NEED  IMMEDIATELY  TO  DEVELOP  DATA  BASES 
DESCRIBING  DIAGNOSTIC  CAPABILITIES  OF  CURRENTLY  FIELDED 
SYSTEMS.  These  data  are  needed  to  support  establishing 
realistic,  achievable  diagnostic  performance  levels. 

(5)  CONTRACT  SPECIFICATIONS  FOR  DIAGNOSTICS  TO  MEET 
OPERATIONAL  CONSTRAINTS  SHOULD  BE  STATED  IN  TERMS  OF  (A)  IDEN¬ 
TIFICATION  OF  SAFETY  AND  MISSION  CRITICAL  FUNCTIONS,  (B)  FALSE 
ALARM  RATE,  AND  (C)  CONSTRAINTS  ON  THE  USE  OF  TEST  EQUIPMENT. 

More  specifically: 

(a)  Safety  and  mission  critical  functions  that  need 
to  be  continually  monitored  for  failure  need  to  be  considered 
in  the  design  effort. 

(b)  The  false  alarm  rate,  which  is  defined  as  per¬ 
centage  of  operator  reported  failure  indications  that  cannot  be 
confirmed  by  maintenance  personnel,  needs  to  be  considered  both 
in  the  design  and  test  approach.  The  performance  level  to  be 
achieved  need  not  be  established  at  the  start  of  the  develop¬ 
ment  process  (see  recommendation  3).  (There  was  disagreement 
among  the  workshop  participants  as  to  whether  the  rate  should 
be  specified  at  some  finite  level  or  zero.  This  area  needs 
further  research.) 

(c)  Constraints  on  the  use  of  external  test  equip¬ 
ment  for  on-system  maintenance  need  to  be  considered  in  the 
mechanization  approach  for  diagnostics. 

(6)  CONTRACT  SPECIFICATIONS  FOR  DIAGNOSTICS  TO  MEET  MAIN¬ 
TENANCE  CONSTRAINTS  SHOULD  BE  STATED  IN  TERMS  OF  (A)  "100  PER¬ 
CENT  DIAGNOSTIC  CAPABILITY,"  INCLUDING  (B)  THE  PERCENTAGE  OF 
THIS  CAPABILITY  THAT  I L  L  BE  AUTOMATIC  AND  MANUAL,  AND  (C)  THE 
ASSOCIATED  SYSTEM  REPAIR  TIMES  SEPARATELY  SPECIFIED  FOR  AUTO¬ 
MATIC  AND  MANUAL  COMPONENTS.  ADDITIONAL  TERMS  THAT  SHOULD  BE 
SPECIFIED  ARE  (D)  UNNECESSARY  REMOVAL  RATE,  AND  (E)  PERSONNEL 
SKILLS. 
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More  specifically: 


(a)  "100  percent  diagnostic  capability."  Specifica¬ 
tions  should  require  the  contractor  to  develop  and  provide  a 
fault  detection  and  fault  isolation  (FD/FI)  capability  that 
addresses  all  incidents  requiring  a  maintenance  action.  This 
would  include  all  types  of  incidents  requiring  maintenance 
personnel  intervention  including  correction  of  failures  (i.e.} 
material  failures  or  out  of  tolerance  conditions,  and/or  veri¬ 
fication  that  no  failure  had  occurred).  In  addition,  the  100 
percent  diagnostic  capability  applies  to  all  items  that  make 
up  a  system  (i.e.,  black  boxes,  wiring). 

(b)  The  diagnostic  capability  should  be  expressed  in 
terms  of  the  percentages  of  the  capability  that  will  be  satis¬ 
fied  automatically  (BIT)  and  manually.  However,  the  specific 
percentages  of  automatic  and  manual  diagnosis  to  be  achieved 
should  not  be  established  arbitrarily  (recommendation  2)  and 
may  be  established  after  the  start  of  the  development  process 
(recommendation  3). 

(c)  System  repair  times  separately  specified  for 
automatic  and  manual  components  of  the  diagnostic  capability. 
These  times  may  be  expressed  either  (individually  or  in  combi¬ 
nation)  as  mean  repair  times,  maximum  allowable  repair  times, 
or  a  repair  time  distribution. 

(d)  Unnecessary  removal  rate.  This  was  defined  as 
the  percentage  of  units  removed  from  the  system  that  are  found 
not  to  contain  a  failure  at  higher  levels  of  maintenance.  The 
specific  rate  to  be  achieved  could  be  established  after  the 
start  of  the  development  process  (recommendation  3)- 

(e)  Personnel  skills.  There  is  a  need  to  go  as  far 
as  possible  in  specifying  skill  levels  in  contracts.  Suggested 
approaches  are  to  specify  the  percentage  of  tasks  to  be  per¬ 
formed  by  high  skilled  and  low  skilled  personnel  or  the  repair 


I 

I 
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times  associated  with  specific  maintenance  actions  for  each 
skill  level. 

(7)  CONTRACTS  SHOULD  INCLUDE  A  SCHEDULE  FOR  MEASURING  THE 
PROGRESS  OF  THE  DESIGN  AND  DEVELOPMENT  OF  BIT.  This  should 
take  the  form  of  subdividing  the  hardware  and  software  and 
requiring  schedule  for  completion  of  BIT  mechanization  for  each 
subdivision.  For  the  software  subdivisions,  completion  would 
mean  a  program  has  been  written,  debugged  and  verified.  Com¬ 
pletion  for  the  hardware  subdivision  should  include  a  paper 
analysis  for  proof  of  design,  the  form  of  which  is  currently 
undefined  (see  Research). 

(8)  THE  CONTRACTOR  AND  PROGRAM  OFFICE  SHOULD  DESIGNATE 
A  SINGLE  ENGINEERING  MANAGER  RESPONSIBLE  FOR  MEETING  THE  100 
PERCENT  DIAGNOSTIC  CAPABILITY.  The  responsibility  should  in¬ 
clude  "vertical  testability,"  that  is,  ensuring  compatibility 
between  on-equipment  fault  detection  and  isolation  levels  and 
tolerances . 

(9)  WHERE  SUBCONTRACTORS  ARE  INVOLVED,  DIAGNOSTIC  RE¬ 
QUIREMENTS  FOR  A  GIVEN  SUBSYSTEM  SHOULD  BE  SELF-CONTAINED  WITH¬ 
IN  THAT  SUBSYSTEM. 

(10)  PLANNING  SHOULD  INCLUDE  DEDICATED  TIME  IN  A  TEST 
FACILITY  OR  A  DEDICATED  TEST  ARTICLE  FOR  BIT  DEVELOPMENT.  Due 
to  the  large  number  and  combination  of  faults  that  could  occur, 
currently  it  is  necessary  to  go  through  an  iterative  test  pro¬ 
cess  to  mature  the  BIT  mechanization.  The  proposed  approach 

is  similar  to  the  reliability  test,  analyze,  and  fix  approach. 
In  the  case  of  complex  weapon  systems,  there  is  probably  a 
need  for  a  dedicated  system. 

(11)  THE  CURRENT  PRACTICE  OF  REQUIRING  CONTRACTUAL  DEMON¬ 
STRATION  BEFORE  DELIVERY  FOR  FIELD  TESTS  OF  THE  BIT  CAPABILITY 
SHOULD  BE  CONTINUED.  This  would  include  verification  that  the 
automatic  diagnostic  objectives  are  being  achieved.  However, 
it  would  be  necessary  to  ensure  that  the  quantitative  BIT  test 
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criteria  are  compatible  with  the  design  criteria.  The  BIT 
design  criteria  are  based  on  100  percent  of  all  maintenance 
incidents  while  the  tests  are  against  a  limited  set  of  inci¬ 
dents  (generally  termed  BIT-addressable  faults).  Further, 
it  is  desirable  to  expand  the  current  demonstration  approach 
to  include  environmental  effects  and/or  a  representative  set 
of  field  failures.  However,  the  techniques  are  not  currently 
available  to  accomplish  this  and  research  must  be  done  to 
expand  the  testing  approach. 

(12)  TESTING  (PARTICULARLY  OPERATIONAL  TESTS)  AND  DATA 
COLLECTION  SHOULD  FOCUS  ON  THE  100  PERCENT  DIAGNOSTIC  CAPA¬ 
BILITY.  Testing  and  data  collector  also  should  evaluate  the 
specified  parameters,  namely,  the  identification  of  critical 
failures,  the  false  alarm  rate,  the  percentage  of  faults 
detected  and  isolated  automatically  or  manually  and  their 
associated  repair  items,  the  unnecessary  removal  rate,  and  the 
adequacy  of  personnel  (need  for  high  skill  personnel)  consider¬ 
ing  all  maintenance  incidents. 

(13)  USE  OF  THE  DIAGNOSTIC  CAPABILITY  THAT  IS  PLANNED  FOR 
FIELD  MAINTENANCE  PERSONNEL  SHOULD  BE  REQUIRED  WHENEVER  THERE 
IS  A  NEED  FOR  SYSTEM  MAINTENANCE.  This  applies  to  maintenance 
performed  by  either  the  contractor  or  the  Service  (user), 
particularly  during  the  development  phase.  A  large  data  base 
is  needed  as  early  as  possible  to  provide  information  for  any 
BIT  development.  Thus,  contractors  should  be  required  to  use 
the  diagnostic  capability  in  acceptance  and  qualification  tests 
and  in  the  manufacturing  and  quality  assurance  processes  to  the 
maximum  extent  possible.  In  addition  to  contributing  to  the 
maturation  of  the  diagnostic  capability,  it  was  suggested  that 
greater  contractor  use  of  diagnostics  in  these  processes  could 
result  in  production  cost  savings. 


(14)  A  PROGRAM  TO  MATURE  THE  DIAGNOSTIC  CAPABILITY  SHOULD 
BE  PLANNED  FOR  THE  EARLY  FIELDED  PRODUCTION  SYSTEMS.  A  two 
year  maturation  program  should  be  planned  for  complex  weapon 
systems  with  extensive  BIT.  This  program  should  include  pro¬ 
visions  for  on-site  collection  of  diagnostic  performance  data 
with  engineering  follow-up  to  provide  corrective  actions. 

C.  RESEARCH  AREAS 

Areas  where  additional  investigation,  analysis,  or 
research  is  required  to  improve  the  specification  and  test 
process  for  diagnostic  capability,  as  well  as  the  design  of 
diagnostic  capability,  are  listed  below. 

(1)  It  is  necessary  to  identify  terms  that  can  be  used  to 
describe  personnel  skill  requirements  in  design  specifications 
and  which  can  be  quantitatively  evaluated  in  test. 

(2)  It  is  necessary  to  develop  the  statistical  methods 
that  should  be  used  for  predicting  and  confirming  of  diagnostic 
system  performance,  particularly  for  BIT. 

(3)  It  is  necessary  to  supplement  the  existing  Maintaina¬ 
bility  Demonstration  Standard  MIL-STD-471  to  include  procedures 
and  environments  that  will  yield  results  more  representative 

of  the  BIT  and  total  diagnostic  capability  that  will  be 
observed  in  the  field.  Approaches  should  be  investigated  for 
selecting  faults  for  the  maintainability  demonstration  that  are 
not  predicated  using  standard  fault  predictions  or  failure 
modes  and  effects  analysis  (perhaps  a  sample  from  operational 
data  on  similar  systems  or  evaluation  by  "independent  verifi¬ 
cation"  similar  to  software). 

(A)  The  use  of  non-volatile  memories  and  memory  inspection 
as  part  of  the  diagnostic  capabilities  should  be  investigated 
to  aid  the  maintenance  technician  in  system  troubleshooting, 
particularly  in  identifying  intermittent  and  environmentally 
related  failures,  and  to  reduce  shop  test  time  and  test  equip¬ 
ment  complexity  at  higher  levels  of  maintenance. 
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(5)  The  type  of  information  that  needs  to  be  displayed  to 
operators  in  the  use  of  BIT  ana  other  diagnostics  aids  should 
be  determined;  especially,  how  *-he  operator  should  be  trained 
to  use  these  data  to  convey  symptoms  to  maintenance  personnel. 

(6)  The  approaches  that  could  be  used  to  specify,  predict 
and  evaluate  false  alarms  and  unnecessary  removals  must  be 
developed . 

(7)  Acquisition  approaches  must  be  developed  to  determine 
contractor  compliance  with  diagnostic  requirements.  Compliance 
should  not  be  limited  to  MIL-STD-^71  demonstration  tests  for 
BIT. 

(8)  The  need  for  an  organization  structure  to  pull 
together  all  of  the  various  activities  (management  procedures, 
technical  training,  T&E,  etc.)  that  are  on-going  in  the  diag¬ 
nostic  area  must  be  investigated.  OSD  was  suggested  as  the 
focal  point  to  organize  and  pull  together  this  structure. 

(9)  A  methodology  for  trade-offs  between  personnel  skill, 
automatic  and  manual  capabilities  and  shop  automatic  test 
equipment  requirements  must  be  developed. 

(10)  A  standard  terminology  for  the  "diagnostic"  area 
should  be  developed. 

(11)  The  implications  of  the  use  of  different  performance 
levels  of  BIT  peacetime  and  wartime  applications,  and  the 
corresponding  manpower  and  other  support  requirements  should 
be  investigated. 
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APPENDIX  A:  BIT  WORKSHOP  PARTICIPANTS 


FOREWORD 


The  BIT  Workshop  on  Requirements  and  Specifications  was 
held  at  the  Institute  for  Defense  Analyses  in  Arlington,  Va. 
on  February  11,  12  and  13,  1981.  The  workshop  was  sponsored 
by  OSD,  MRA&L  (Manpower,  Reserve  Affairs  and  Logistics)  and 
hosted  by  IDA.  Attendence  was  by  invitation  of  OSD  and  was 
limited  to  40  participants  so  as  to  promote  a  free  exchange 
of  ideas  and  experiences.  The  attendees  were  welcomed  to  the 
IDA  facility  by  Dr.  Harry  Williams,  Director,  Program  Analysis 
Division,  IDA.  Mr.  Martin  Meth,  OSD,  introduced 'the  workshop. 
A  total  of  ten  presentations  were  given,  followed  by  meetings 
of  three  separate  Panels  and  subsequent  presentation  of  Panel 
reports . 
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BITE 
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CFE 
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CSMC 
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Aircraft 
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Air  Data  Computer 

Acronym  for  a  Navy  surface-to-air  missile  system 
Avionics  Intermediate  Shop  (I-level  test  equipment) 
Air  Force  Systems  Command 
Air  Force  Test  and  Evaluation  Center 
Aeronautical  Systems  Division  (of  AFSC) 
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Automatic  Test  Equipment 
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Built-In  Test 
Puilt-In-Test  Equipment 
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Central  Processing  Unit 

Communication  System  Control  Set 

Combat  System  Maintenance  Control  (AEGIS) 
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Detect 

DAC 

Digital  to  Analog  Converter 

DEGD 

Degrade 

D-Level 

Depot  Level 

D/L 

Data  Link 

DSPL 

Display 

ECM 

Electronic  Countermeasures 

ECP 

Engineering  Change  Proposal 

EDM 

Engineering  Development  Model 

ENR 

Expected  Number  of  Removals 

E/0 

Electro/Opticai 

EPG 

European  Participating  Governments 

EPI 

Engine  Performance  Indicator 

ET 

Electronics  Technician 

ETE 

External  Test  Equipment 

EU 

Electronic  Unit 

P-18 

Hornet  'Fighter'  Configuration 

FA 

False  Alarm 

FC 

Fire  Control 

FCC 

Fire  Control  Computer 

FCES 

Flight  Control  Electronic  Set 

FCS 

Flight  Control  System 

FCSA(B) 

FCS  Channel  A  (Channel  B) 

FD/FI 

Fault  Detection/Fault  Isolation 

FH 

Flight  Hours 

FLIR 

Forward  Looking  Infrared  Set 

FLU 

Flight  Line  Unit 

FMEA 

Failure  Mode  and  Effects  Analysis 

FOM 

Figure  of  Merit 

FSD 

Full  Scale  Development 

FSE 

Navy  Fleet  Support  Evaluation 

GD 

General  Dynamics 

GFE 

Government  Furnished  Equipment 

GSE 

Ground  Support  Equipment 
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HARM 

HP 

HOL 

HUD 

HSD 

I 

IBS 

ICS 

IECMS 

IEEE 

IFF 

I-Level 

ILS 

INS(U) 

IOP 

IR&D 

JLC 

K 

LBTS 

LRU 

LST/SCAM 

MAD 

MATE 

MC 

MCAIR 

MDC 

M-Demo 

MFL 

MI 

MMD 

MMH 

MMP 

MOT&E 


High  Speed  Anti-Radiation  Missile 

High  Frequency 

Higher  Order  Language 

Head  Up  Display 

Horizontal  Situation  Display 

Isolate 

Interference  Blanker  Set 
Intercom  Set 

Integrated  Engine  Condition  Monitoring  System 
Institute  for  Electrical  &  Electronic  Engineers 
Identification — Friend  or  Foe 
Intermediate  Level 

Integrated  Logistics  Support  (also  Instrument 
Landing  System) 

Inertial  Navigation  System  (Unit) 

Input/Output  Processor 
Internal  R&D 

Joint  Logistics  Commanders 
Kilo  (1000) 

Land  Based  Test  Site  (AEGIS) 

Line  Replaceable  Unit 

Laser  SPA  Tracker/Strike  Camera  Pod 

Magnetic  Azimuth  Detector 

Modular  ATE 

Mission  Computer 

McDonnel]  Aircraft  Company 

McDonnell  Douglas  Corporation 

Maintainability  Demonstration 

Maintenance  Fault  List 

Memory  Inspect 

Cockpit  Master  Monitor  Display 
Maintenance  Manhours 
Maintenance  Monitor  Panel  (M  Panel) 
Multinational  OT&E 
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MRA&L  Manpower,  Reserve  Affairs  ar.d  Logistics  (OSD) 

MTBF  Mean  Time  Between  Failure  (hours) 

MTBR  Mean  Time  Between  Removal  (hours) 

MTTR  Mean  Time  to  Repair  (hours) 

MUX  Multiplex 

MUX  BUS  Avionic  Digital  Multiplex  Bus 

NABIT  Built-in-Test  for  Nonavionic  Subsystems 

NAC  Naval  Avionics  Center 

NAVAIR  Naval  Air  Systems  Command 

NAVMAT  Naval  Material  Command 

NAVSEA  Naval  Sea  Systems  Command 

NEC  Naval  Engineering  Center 

NOSC  Naval  Ocean  Systems  Center 

NPE  Navy  Preliminary  Evaluation 

NSWC  Naval  Surface  Weapons  Center 

OFP  Operational  Flight  Program 

0-Level  Organizational  Level 

ORTS  Operational  Readiness  Test  System  (used  with  AEGIS) 

OSD  Office  of  the  Secretary  of  Defense 

OT&E  Operational  Test  and  Evaluation 

Pto  BIT  Performance  Fault  Threshold 

RADC  Rome  Air  Development  Center 

RALT  Radar  Altimeter  Set 

R&R  Remove  and  Replace 

RCVR  Receiver 

RDDI/LDDI  Right/Left  Cockpit  Digital  Display  Indicator 

RETOK  (also  RTOK)  Retest  Okay  (i.e.,  without  removal 

from  operational  installation 

Restrt  Restart  BIT  Test 

RIU  Remote  Interface  Unit 

RMRB  Reliability  Maintainability  Review  Board 

SDRS  Signal  Data  Recorder  Set 

SE  Support  Equipment 

SEM  Standard  Electronic  Modules 
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S-Level  Shop  Level 

Sixty-1  (Also  60-1;  an  Air  Force  maintenance  reporting 
system 

SMS  Stores  Management  Set 

SPO  System  Program  Office 

SRA  Special  Replacement  Assembly 

SRU  Shop  Replaceable  Unit 

ST/BIT  Self  Test/Built-in  Test 

STFF  Self  Test  Fault  Flag 

TACAN  Tactical  Air  Navigation 

TON  TACAN  Set 

T.O.  Technical  Order 

TRA  Test  Requirements  and  Analysis 

At  Maintenance  Time  Saved 

s 

UFC  Cockpit  Up-Front  Control  Panel 

USOR  Utility  to  Save  and  Restore  the  Disk 

V  Verify 

VSWR  Voltage  Standing  Wave  Ratio 

WRA  Weapons  Removable  (Replaceable)  Assembly 


OPENING  REMARKS 


MARTIN  METH,  OSD,  MRA&L 

Mr.  Meth  is  the  Staff 
Engineer  for  OSD, 
MRA&L 


The  agenda  for  the  BIT  workshop  is  shown  as  Figure  1-1. 

Case  studies  and  associated  discussions  are  scheduled,  followed 
by  panel  discussions  and  reports. 

The  scope  of  the  workshop  involves  the  areas  of  require¬ 
ments  for  built-in-test  and  diagnostics,  and  the  methods  of 
testing  to  ensure  that  the  requirements  have  been  met.  The 
specific  objectives  are  to  characterize  current  practices  and 
results,  to  recommend  improvements  and  methods  for  implementa¬ 
tion,  and  to  identify  those  areas  where  research  is  required. 

Figure  1-2  is  an  extract  from  an  Air  Force  report  outlining 
the  past  history  of  many  BIT  equipments  in  which  specific  diag¬ 
nostic  requirements  have  not  been  met  and  also  indicating  the 
absence  of  specified  values  for  certain  parameters.  This 
information  indicates  that  there  is  much  improvement  to  be  made 
in  the  areas  of  specifying,  verifying  and  testing  BIT  and 
diagnostics . 

The  process  of  system  development  is  shown  in  Figure  1-3; 
the  overall  areas  addressed  in  this  workshop  are  circled  in  the 
Figure.  The  topics  to  be  addressed  by  each  of  the  three  panels 
are  shown  in  Figures  1-4,  1-5  and  1-6.  Reports  on  these  topics 
(and  others  as  appropriate)  will  be  presented  to  the  full  panel 
and  other  invited  attendees  at  the  conclusion  of  the  workshop. 


3 (JILT- IN-TEST  (BIT)  REQUIREMENTS  AND 
EVALUATIONS  WORKSHOP 


WEDNESDAY,  FE3.  11,  1981 


8:30 

9:00 

Check-In 

9:00 

- 

9:30 

Introduction  &  Announcements 

Martin  Meth 

9:30 

- 

10:00 

Background  -  NAVMAT 

George  Neumann 

10:00 

- 

10:15 

3reak 

10:15 

- 

12  :00 

Operational  Testing  -  ARTEC 

Maj.  Vince  Linden 

12:00 

- 

1:00 

Lunch 

1:00 

- 

2:15 

?16  -  General  Dynamics 

Gordon  England 

2:15 

- 

3:30 

?16  PCS  -  Westingnouse 

Roy  Pyle/Jim  Victor 

3:30 

- 

3:45 

3reak 

3:45 

- 

5:15 

F18  -  McDonnell-Douglas 

3ob  Drummond/ 

Ed  Meyer 

5:15 

6:30 

6:00 

AYK-14  -  CDC 

Dinner,  Ft.  Myer 

Wei  Long  Chen 

THURSDAY 

,  FEB. 

12,  1981 

8:00 

8:30 

Check-In 

Martin  Meth 

8:30 

- 

9:00 

ALQ-126  -  Maint.  Tech. 

Ken  Wilson 

9:00 

- 

9:30 

SPS-67  -  NOSC/NAC 

Mel  Nunn/Johr.  Rogers 

9:30 

- 

10:15 

AEGIS/GRTS  -  RCA 

Howard  Boardman 

10 : 15 

- 

10:30 

Break 

10:30 

12:00 

BIT  Studies  -  RADC 

Tony  Coppola/ 

Capt.  ban  Gleason 

12:00 

- 

1:00 

Lunch 

1:00 

- 

1 :  3C 

Instructions  to  Panels 

Martin  Meth 

1:30 

- 

5:00 

Panel  Discussions 

Panels 

5:00 

- 

6  : 00 

General  Panel  Recap 

Panel  Chairman 

FRIDAY, 

FEB.  13, 

1981 

8:30  - 

9:00 

Check-In 

Martin  Meth 

9:00  - 

10:45 

Panel  Reports 

Panel  Chairmen 

10:45  - 

11:00 

Summary  of  Results 

Martin  Meth 

Figure 


TOE  FELAT10NSHIPS  BEMEN  BIT  PERDFFWKCE  AND  (FERATiCNAL  NEEDS 
FOR  LOGISTICS  AND  MANPOWER  EQUIPMENTS  APE  NOT  WELL 
UNDERSTOOD. 


INADEQUATE  FESOURCES  AND  UNREALISTIC  SCHEDULES  APE  KING 
PROPOSED  FOR  DEVELOPMENT  OF  BIT. 


GROUP  32.  SUBSYSTEM  SPECIFICATION  AND  TESTING 

TOE  RESULTS  OF  CONTRACTOR  BIT  DEMONSTRATIONS  (USING  FAULT  INSERTIONS) 

DOES  NOT  MATCH  BIT  PERFOFMME  IN  TOE  FIELD  Figure  1-5 

ANALYSIS  OF  BIT  DESIGN  EFFORTS  ARE  NOT  PROVIDING  DMA  TO  DETERMINE 
IF  BIT  SPECIFICATIONS  CAN  BE  MET. 

FUNDING  AND  SOtIULE  ALLOCATED  FOR  BIT  IIVELCRENT  AIT  &NERALLY 
NOT  ADEQUATE. 


GROUP  «3,_ SYSTEM  REQUIREMENTS  AND  TESTING 


Figure  1-6 


PERFORMANCE  OF  BIT  IN  TOE  FIELD  IS  GENERALLY  MUCH  LOWER  THAN  OBSERVED 
IN  TESTING  PRIOR  TO  PRODUCTION. 

CURRENT  APPROACHES  TO  BIT  SYSTEM  DESIGN  DO  NOT  TAKE  INTO  ACCOUNT 
«W  REAL  WORLD  PRCBLEMB  AS  EVIDENCED  BY  HIGH  LEVELS  OF  FALSE 
ALARMS,  UNDETECTED  FAILURE,  RETGT  OK'S  AND  AMBIGUITIES. 


BIT  PERFORMANCE  IN  TOE  FIELD  HAS  NOT  BEEN  COMPATIBLE  WITH  PUWED 
PERSONAL  SKIUS,  TEST  EQUIPMENT,  AND  OTOER  LOGISTICS. 


FUNDING  AND  SCHEDULE  ALLOCATED  FOR  BIT  DEVELOPMENT  ARE  GENERALLY 
NOT  ADEQUATE 
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BIT  PROGRAMS 


George  Neumann 
NAVMAT-04T 


Mr.  Neumann  is  the  technical 
director,  test  and  monitoring 
systems  program  office  of 
NAVMAT . 


HIGHLIGHTS  OF  THE  PRESENTATION 

The  primary  objective  of  the  various  3IT  programs  is  to 
develop  improved  methods  of  specifying  and  evaluating  BIT. 
Recent  relevant  industry  and  service  efforts  addressing  this 
objective  are  shown  in  Figure  2-1.  The  significant  points 
are — 


»  These  programs  have  brought  the  BIT,  testability,  and 
logistics  problems  to  the  attention  of  high  level 
military  and  civilian  personnel  in  the  Department  of 
Defense . 

o  BIT  is  viewed  as  a  subset  of  testability. 

The  Navy  BIT  Workshop  was  held  in  December  1980  and  in¬ 
cluded  as  particioants :  PMA  265  (F/A-18);  OPTEVOR;  NSIA  Ad 
Hoc  Automatic  Testing  Group;  and  the  Navy  Testing  Technology 
Team.  A  report  of  the  results  of  this  workshop  is  available 
from  NAVMAT-O^T.  The  primary  Navy  BIT  Workshop  findings  are 
shown  in  Figure  2-2.  Significant  points  are — 

<*  The  BIT  implementation  problem  is  a  management  problem 
since  technology  is  available. 

•  BIT  requirements  should  reflect  operational 
requirements . 

•  It  is  not  realistic  to  prescribe  across-the-board 
rates  for  fault  detection,  fault  isolation,  and  false 
alarms . 

•  Advantage  should  be  taken  of  knowledge-based  systems. 
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Figure  2-3  presents  a  comparison  of  management-oriented 
recommendations  which  are  the  result  of  the  Navy  Ad  Hoc  Project, 
the  Industry/Joint  Services  Automatic  Test  Project  and  the  Navy 
BIT  Workshop  (December  1980),  and  lists  those  areas  where  such 
recommendations  are  being  implemented  by  on-going  efforts.  An 
examination  of  Figure  2-3  indicates  the  absence  of  current 
efforts  in  certain  areas  that  have  been  recommended  for  action. 
Significant  points  are — 

•  The  Joint  Service  BIT  Design  Guide  is  to  be  reissued. 

•  Design  for  Testability  Course  is  a  two  to  three  day 
course  presented  by  NSV/C  for  designers  of  weapons 
systems . 

Figure  2-4  presents  a  comparison,  among  the  three  BIT  Programs, 
of  recommended  tools  for  implementing  testability  requirements 
together  with  a  statement  of  current  efforts  in  these  areas. 
Current  efforts  are  underway  in  all  areas  addressed.  Finally, 
Figure  2-5  presents  a  comparison  of  technical  recommendations. 
There  are  no  current  efforts  underway  in  the  areas  of  vertical 
testability,  "smart"  BIT,  and  BIT  calibration.  It  is  signifi¬ 
cant  to  note  that  the  time  required  in  operational  use  to 
"mature"  the  BIT  (i.e,,  to  tailor  the  BIT  to  an  operational 
environment  and  to  utilize  it  most  effectively)  may  be  two  to 
three  years . 

A  summary  of  the  requirements  of  an  effective  BIT  program 
is  as  follows: 

A  BIT  program  is  required  which  tracks  the  weapon  system 
acquisition  cycle  and  includes — 

•  Management  and  user  involvement  including  realistic 
specification,  closed-loop  tracking  and  reporting; 

•  Continued  development  of  standards,  specifications, 
guides  and  tools; 

•  Technology  development. 


$ 


DISCUSSION  POINTS 

•  Experience  with  fielded  systems  has  indicated  a  BIT 
false  alarm  rate  between  20  to  30  percent  in  RADC 
studies . 

•  Other  RADC  studies  involving  nine  different  Air  Force 
systems  at  numerous  bases  have  shown  unnecessary  re¬ 
moval  rates  on  the  order  of  ^0  percent  with  some  sys¬ 
tems  as  high  as  89  percent. 

•  Multiple  "box-swapping"  maintenance  practices  are  used 
in  the  field  because  of  shortcomings  in  BIT,  according 
to  RADC  studies  of  Air  Force  systems. 

•  False  alarms  are  sometimes  defined  as  "something  the 
operator  learns  to  ignore." 

•  False  alarms  may  indicate  a  true  fault  that  does  not 
require  immediate  correction. 

•  False  alarms  may  indicate  degradation  trends,  particu¬ 
larly  if  their  interarrival  periods  are  decreasing. 

®  False  alarms  may  indicate  intermittent  failures  as 
shown  by  CND  (could  not  duplicate)  and  RETOK  (retest 
okay)  rates. 

•  Airlines  find  that  far  less  than  50  percent  of  boxes 
removed  contain  verified  failures,  especially  auto¬ 
pilots  (the  worst)  which  run  85  to  90  percent  non- 
verified . 

•  The  increased  rate  of  false  alarms  in  the  autopilot 
systems  probably  reflects  the  criticality  of  this  sys¬ 
tem  ana  the  fact  that  it  is  more  heavily  monitored  by 
BIT  than  other,  less  critical,  systems. 

•  Another  problem  with  autopilots  is  CND's  on  the  ground, 
partly  because  of  the  inability  to  reproduce  the  flight 
environment  on  the  ground  (e.g.  "porpoising"  in  the 
pitch  axis).  Many  unnecessary  replacements  of  the 
pitch  computer  result . 

•  Autopilot  "squawks"  are  often  identified  by  pilots  as  a 
defect  in  a  major  unit  that  may  not  be  at  fault. 

•  BIT  systems  up  to  now  have  been  widely  ignored  by  air¬ 
line  maintenance  people  because  01  the  lac..:  of  agree¬ 
ment  between  BIT  fault  indications  and  flight  crew 
"squawks . " 

•  Airlines  are  requiring  BIT  in  new  systems  to  incor¬ 
porate  fault  diagnostic  memories  to  address  intermit- 
tents,  with  user  options  as  to  how  many  flighc  legs 
are  recorded  along  the  lines  of  "smart"  BIT. 
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•  The  airlines  are  asking  that  in  future  complex  systems 
BIT  be  implemented  using  a  master  LRU  with  an  alpha¬ 
numeric  readout  identifying  the  failed  unit  and  its 
future  mode. 

•  The  need  for  rapid  turnaround  time  for  the  airlines 
results  in  many  unnecessary  removals  and  replacements 
of  units.  This,  in  turn,  results  in  the  requirement 
for  increased  spares.  BIT  presently  has  little  effect; 
on  this  because  it  is  generally  ignored. 

•  Airline  pilots  are  often  drivers  of  unit  replacements 
(possibly  unnecessarily)  because  they  insist  that  the 
unit  be  replaced,  particularly  in  radar  systems. 

•  The  new  NAVAIR  maintainability  standard  presently  in 
progress  contains  certain  numerical  requirements. 

This  approach  must  be  reviewed  carefully  before  pub¬ 
lishing  the  standard. 
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Figure  2-1 

RELEVANT  INDUSTRY/SERVICE  EFFORTS 


o  INDUSTRY  AD  HOC  ATE  PROJECT  FOR  THE  NAVY 
o  INDUSTRY/JOINT  SERVICES  AUTOMATIC  TEST  PROJECT 
o  AFTEC  STUDY  (AUTOMATIC  DIAGNOSTIC  SYSTEMS) 
o  NAVY  BIT  MINI-WORKSHOP 
o  JLC/NAVY  CURRENT  EFFORTS 


Figure  2-2 

NAVY  WORKSHOP  FINDINGS 


•  TECHNOLOGY  EXISTS:  BIT  DEVELOPMENT  IS  A  MANAGEMENT  PROBLEM 

•  BIT  REQUIREMENTS  SHOULD  REFLECT  OPERATIONAL  REQUIREMENTS:  ACROSS 
THE  BOARD  OR  STANDARD  FD/FI/FAR  PERCENTAGES  NOT  A  REALISTIC  APPROACH 

•  BIT  ALLOCATION  IS  INFLUENCED  BY  OTHER  FACTORS  {E.G.:  RELIABILITY, 
MAINTAINABILITY,  MANNING) 

•  BIT  DEVELOPMENT  IS  AN  ITERATIVE  PROCESS 

©  ADAPTIVE  DESIGN  IS  LAGGING:  "SMART"  BIT  NEEDS  TO  BE  DEVELOPED 
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TRENDS  AND  FUTURE  APPROACHES  IN  AUTOMATED  DIAGNOSTICS 


Major  Vince  Linden 
AFTEC 

Major  Vince  Linden  is  the 
OT&E  Logistics  Assessment 
Policy  Manager  for  AFTEC. 


HIGHLIGHTS  OF  THE  PRESENTATION 

The  purpose  of  the  presentation  is  to  discuss  trends 
towards  highly  automated  diagnostics  for  maintenance  personnel 
in  tie  field,  recent  OT&E  results,  and  impacts  on  planned 
maintenance  and  training  concepts;  and  to  discuss  future  ap¬ 
proaches  towards  the  acquisition  of  diagnostic  systems.  The 
approach  used  to  achieve  this  purpose  is  through  presentation 
of  BIT  background  information,  discussion  of  diagnostic  theory 
and  a  display  of  E-3A,  F-16  and  EF-111A  BIT  test  results. 

BIT  trends  in  the  1970s  in  terms  of  personnel  and  systems 
are  shown  in  Figure  3-1.  Training  and  maintenance  concepts 
and  the  expected  results  based  on  the  trends  of  the  1970s  are 
shown  in  Figures  3-2  and  3-3-  First  term  productivity  means 
"first  enlistment.’’)  However,  because  of  inadequate  perform¬ 
ance  of  BIT,  the  actual  impacts  are  as  shown  in  Figures  3— ^ 
and  3-5<  The  increa..  ?s  shown  are  those  above  the  planned 
requirements,  due  primarily  to  the  fact  that  the  systems  (in¬ 
cluding  BIT)  are  not  mature.  The  problems  with  OT&E  as  indi¬ 
cated  in  Figure  3-5  derive  from  the  fact  that  the  systems  have 
not  been  tested  by  the  contractor  as  the  system's  diagnostics 
would  be  used  in  an  operational  environment.  In  many  cases, 
Tech  Orders  (Manuals)  are  not  available  for  use  in  maintenance 
by  the  contractor.  Use  of  immature  diagnostics  slow  down 
contractor's  testing  and  cost  him  time  and  money.  In  addition 
the  government  has  not  yet  been  able  to  properly  articulate 
diagnostics  requirements  to  contractors. 
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Improved  methods  for  use  of  diagnostics  have  been  applied 
to  the  E-3A  and  subsequently  enhanced  for  application  to  the 
P-16  and  EF-11A  as  indicated  in  Figure  3-6.  The  two  types  of 
systems  used  in  the  F-l6  to  which  BIT  is  applied  are  shown  in 
this  Figure.  The  flight  control  system  is  primarily  an  analog 
system  whereas  the  "MUX  BUS"  is  a  digital  system. 

Fault  detection  and  isolation  requirements  are  shown  in 
Figure  3-7.  These  can  be  met  by  incorporating  100  percent 
fault  detection  in  units  1  through  5  and  100  percent  fault 
isolation  in  units  1  through  4.  Breakdowns  between  automatic 
and  manual  fault  detection  and  isolation  are  as  shown.  The 
numbers  shown  must  be  precisely  defined  since  they  can  mean 
different  things  to  the  contractor  and  the  government.  No 
values  have  been  shown  for  CNDs,  RETOKs,  and  false  alarms. 

The  term  "CND"  is  used  to  identify  the  situation  where  0-level 
maintenance  is  not  able  to  reproduce  the  "squawk"  reported  by 
the  flight  crew  on  BIT.  The  term  "RETOK"  is  used  to  identify 
the  situation  where  0-level  maintenance  has  verified  the 
reported  fault  but  I-level  maintenance  has  tested  the  unit  and 
found  no  fault.  In  this  case,  the  problem  was  apparently 
solved  by  the  replacement  of  the  "squawked"  unit.  Repair  times 
(MTTR)  need  to  be  specified  for  both  automatic  and  manual  modes 
as  well  as  the  percentage  of  actions"  in  each  mode.  Special 
category  data  (or  events)  are  defined  as  non-addressable  events 
and  have  not  been  included  for  this  analysis;  specifically 
these  category  data  include  human  error,  other  maintenance, 
cannibalizations,  deferred  maintenance,  interrelated  failures, 
and  BIT/FIT-pertinent  data  discovered  during  trouble  shooting. 

It  is  important  to  know  the  elements  of  MTTR  in  order  to 
focus  on  the  problem  of  high  MTTR  and  unsatisfactory  turnaround 
time.  In  some  cases,  systems  in  the  field  have  horrible 
malntanability  problems  but  the  maintenance  people  are  keeping 
the  system  up  at  an  excessively  high  cost  in  resources.  For 


example,  one  LRU  in  the  E-3A  requires  a  hoist  and  pulley  system 
to  remove  it  from  the  aircraft,  resulting  in  a  very  high  remove/ 
replace  time. 

As  BIT/FIT  becomes  more  successful,  the  fault  detection 
isolation  portion  of  MTTR  becomes  a  smaller  and  smaller  per¬ 
centage  of  MTTR.  However,  when  BIT/FIT  is  not  successful,  the 
"beyond  BIT"  maintenance  becomes  increasingly  more  difficult 
and  requires  higher  skill  level  personnel.  Properly  trained 
people  and  adequate  T.O.s  are  often  not  available  to  provide 
such  maintenance  when  it  is  needed.  In  addition,  different 
BIT/FIT  systems  will  differ  in  diagnostic  characteristics  ac¬ 
cording  to  definitional  differences,  mechanization  and  system 
design.  In  some  cases,  maintenance  policies  will  also  mask  the 
true  diagnostic  capability. 

The  data  base  used  to  obtain  E-3A  test,  results  is  shown 
in  Figure  3-8;  E-3A  radar  events  are  shown  in  3-9;  and  radar 
maintenance  events  are  shown  in  Figures  3-10  and  3-11.  Only 
the  surveillance  radar  of  the  E-3A  is  included  in  this  analy¬ 
sis.  Certain  false  alarms  are  presently  being  tracked  for 
possible  trend  analysis  to  detect  degradation  in  unit  perform¬ 
ance.  Fault  isolation  in  the  E-3A  is  off-line,  requiring 
transfer  of  the  diagnostic  programs  for  execution.  Addressable 
events  are  shown  in  Figure  3-12,  indicating  the  effect  of  CNDs 
on  detection  and  non-detection  percentages  as  viewed  by  the 
government  and  the  contractor;  and  the  effect  of  RETOKs  on 
isolation  percentage  as  viewed  by  the  government  and  the  con¬ 
tractor.  As  can  be  seen,  CND  and  RETOK  requirements  were  not 
imposed  on  the  contractor  for  the  E-3A.  The  actual  auto¬ 
isolation  percentage  is  shown  to  be  between  3^  and  ^9  percent. 
Some  of  the  RETOKs  were  actually  bad  but  no  problem  could  be 
found  at  the  repair  facility.  The  RETOK  rate  was  essentially 
the  same  whether  fault  isolation  was  automatic  or  manual  (about 
30  percent). 
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The  AFTEC  tests  were  started  two  years  after  the  E-3A  had 
been  in  the  field.  The  test  was  condulted  by  the  Air  Force 
with  the  assistance  of  Boeing  and  Westinghouse  Engineering 
support  personnel.  The  data  are  considered  highly  accurate 
since  they  were  checked  by  Air  Force  Test  Team  personnel  prior 
to  entry  into  the  data  base. 

The  original  E-3A  spec  requirements  of  98  percent  auto¬ 
matic  fault  detection,  95  percent  automatic  fault  isolation 
were  reduced  to  90/79  percent  respectively. 

The  impact  to  the  E-3A  as  a  result  of  inadequate  diag¬ 
nostics  are  shown  in  Figure  3-13-  In-flight  repair  was  not 
evaluated  during  the  test  period  since  only  21  attempts  at  in¬ 
flight  repair  were  documented  and,  of  these,  only  two  were 
successful.  Spares  are  carried  on-board  the  E-3A  to  effect  in¬ 
flight  repair.  However,  the  radar  must  be  shut  down  for 
repair  actions  and  this  was  usually  not  done  due  to  mission 
impacts.  Maintainability  deficiencies  have  been  largely  over¬ 
come  due  to  extraordinary  supply  measures,  employed  by  the  552 
AWACW  and  the  E-3A  system  manager.  Additional  TOs  (manuals) 
have  been  required  both  for  the  system  ar.d  its  support 
equipment . 

Extensive  additional  training  requirements  have  been 
experienced  and  documented  by  the  Wing.  As  a  result,  the  Air 
Force  must  track  individual  personnel  qualifications  for 
assignment  to  various  E-3A  units. 

The  data  base  used  to  obtain  the  F-l6  test  results  are 
shown  in  Figure  3-1*1.  The  characteristics  of  the  two  differ¬ 
ent  types  of  systems  (flight  control  and  MUX-BUS)  are  shown 
in  Figure  3-15-  Several  comments  are  in  order:  Flight 
control  data  (fly-by-wire,  totally  electronic)  are  not  polluted 
by  data  from  interactions  among  systems.  The  MUX-BUX  incor¬ 
porates  many  systems.  Fault  detections  in  the  flight  control 
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are  presented  to  the  pilot  who  must  record  the  fault  indica¬ 
tions,  as  opposed  to  the  MUX-BUS  which  records  the  faults  in 
memory  for  later  readout.  As  a  result,  some  of  the  faults 
'particularly  intermittents )  are  missed  in  the  analog  flight 
control  system. 

Figure  3-16  shows  the  box  score  reflecting  the  ST'/BIT 
(self-test,  built-in-test)  performance  of  the  F-16  flight 
control  system  (a  quad-redundant  system),  as  seen  by  the  con¬ 
tractor  and  by  the  user.  Operational  experience  indicates 
that  both  reliability  and  maintaj nability  are  improved  the 
more  aircraft  is  flown. 

Figure  3-17  shows  a  similar  box  score  for  the  F-l6  MUX  BUS. 
The  fault  reporting  on  the  MUX  BUS  does  not  include  failures  of 
input  devices  to  the  units  on  the  MUX  BUS  (e.g.,  angle  of 
attack  unit  as  an  input  to  the  CADC).  Correction  of  this 
deficiency  requires  imposing  testability  requirements  on  the 
units  (e.g.,  CADC)  and  impacts  requirements  for  GFE. 

The  impacts  of  F-l6  ST/BIT  shortfalls  or  inadequate  diag¬ 
nostics  point  to  several  problems:  There  is  a  need  for  addi¬ 
tional  trouble-shooting  guides;  the  training  courses  need 
restructuring;  there  appear  to  be  Avionics  Intermediate  Test 
Station  (AIS)  compatability  problems;  and  many  support  deci¬ 
sions  are  still  uncertain. 
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Figure  3-^8  shows  the  extent  of  the  EF-111A  test  effort, 
a  modest  effort  compared  to  the  E-3A  and  the  F-16.  Figures 
3-19  and  3-20  indicate  the  results  of  the  testing.  The 
impacts  of  the  EF-111A  BIT/BITE  are  too  early  to  determine. 

This  system  will  be  tested  in  depth  during  the  October-December 
81  timeframe . 

Conclusions  and  recommendations  are  shown  in  Figures  3-21 
through  3-26.  General  conclusions  are  that  user-command  per¬ 
sonnel  must  become  more  involved  in  defining  deployment7 
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employment  concepts  and  operational  and  maintenance  constraints, 
and  in  defining  his  requirements.  One  hundred  percent  diagnostic 
capability  is  required  using  both  automatic  and  manual  diagnostics 


BACKGROUND 


Figure  3-1 

TRENDS  IN  THE  1970s 


•  PERSONNEL  INSTABILITY  -  HIGH  TURNOVER 

•  TRAINING  COST  INCREASES 

•  FIRST  TERM  PRODUCTIVITY  INCREASE  NEEDED 

•  OPERATIONAL  READINESS  IMPROVEMENTS  NEEDED 

•  LIFE  CYCLE  COST  REDUCTION  EMPHASIZED 

•  TECHNOLOGICAL  ADVANCES 

-  MICRO  PROCESSORS 

-  “SMART”  DIAGNOSTIC  SYSTEMS 

•  HEAVY  INVESTMENT  IN  SOPHISTICATED  BIT 

•  TRAINING  AND  MAINTENANCE  CONCEPTS  DEVELOPED  AROUND  BIT 


Figure  3-2 


BACKGROUND 


BIT  TRAINING  CONCEPT 


•  ASSUMPTIONS 

-  BIT  CAN  ELIMINATE  NEED  TO  TEACH  SYSTEM  THEORY 

-  LONG  TECHNICAL  SCHOOLS  DRIVE  TRAINING  COSTS 

-  OPERATIONAL  COMMANDS  NEED  GREATER  FIRST  TERM  PRODUCTIVITY 

•  ACTIONS 

-  PROVIDING  ONLY  TASK  ORIENTED/ BIT  DIRECTED  TRAINING 

-  DOING  MAXIMUM  TRAINING  AT  FIRST  OPERATIONAL  BASE 

•  EXPECTED  RESULTS 

-  REDUCED  FIRST  TERM  FORMAL  SCHOOLING 

-  INCREASED  FIRST  TERM  PRODUCTIVITY 
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BACKGROUND 


B«T  MAINTENANCE  CONCEPT 


•  ASSUMPTIONS 

—  BIT  CAN  REDUCE  TROUBLE  SHOOTING  TIME 

-  BIT  CAN  REDUCE  SUPPORT  EQUIPMENT  REQUIREMENTS 

•  ACTIONS 

-  TECHNICAL  DATA  WRITTEN  AROUND  BIT  OPERATION 

-  SUPPORT  EQUIPMENT  NOT  PROCURED 


•  EXPECTED  RESULTS 

-  REDUCED  MANNING 

-  INCREASED  OPERATIONAL  READINESS 

-  REDUCED  LIFE  CYCLE  COST 


Figure  3-4 

PROBLEM  WITH  THE  CONCEPTS 


BACKGROUND 


•  BIT  SYSTEMS  FAILED  TO  PERFORM  ADEQUATELY 

•  IMPACTS 

-  EXTENSIVE  ADDITIONS  TO  TECH  DATA 

-  ADDITIONAL  TRAINING 

~  SUPPLEMENTAL  SUPPORT  EQUIPMENT 
~  MORE  TECHNICIANS 

-  CONTINUING  CONTRACTOR  SUPPORT 

-  +$ 
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Figure  3-5 

OT&E  EXPERIENCE 


BACKGROUND 


•  INCOMPLETE,  IMMATURE  DIAGNOSTICS 

•  RARELY  USED  BY  CONTRACTOR 

•  DIAGNOSTIC  SPECIFICATIONS  NOT  MEANINGFUL 

•  IMPROVED  EVALUATION  METHODOLOGY  NEEDED 


Figure  3-6  DIAGNOSTIC  THEORY 
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DETECTION 


Figure  3-7 


DIAGNOSTIC  THEORY 


THEORY  FOR  DESIGNING  A  90/80% 
FD/Fi  CAPABILITY 
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E-3A  TEST  RESULTS 


TEST  RESULTS 


•  DATA  BASE 

- 19  AIRCRAFT 

-  18  MONTHS  (JUL  78  •  DEC  79) 

-  791  SORTIES 

-  6,205  FLYING  HOURS 

•  DEDICATED  BIT/FIT  TEST  TEAM 

•  TINKER  AFB  OK 


a-9-81-7 


22 


Figure  3-9 


TEST  RESULTS 
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Figure  3-10 

E-3A  RADAR  MAINTENANCE  EVENTS 
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Figure  3-11 


E-3A  RADAR  EVENTS 
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E-3A  RADAR  E5T/FIT  PERFORMANCE 
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Figure  3-13 

E-3A  BIT/FIT  SHORTFALL  IMPACTS 


TEST  RESULTS 


•  INFLIGHT  REPAIR  NOT  USED 

•  18  ADDITIONAL  RADAR  TOs  DEVELOPED 

•  9  ADDITIONAL  SUPPORT  EQUIPMENT  TOs  DEVELOPED 

•  ADDITIONAL  SUPPORT  EQUIPMENT  NECESSARY 

•  6  ADDITIONAL  TRAINING  MONTHS  (6  COURSES) 

•  25  ADDITIONAL  MONTHS  IN  OJT  REQUIRED 

Figure  3— 1 ^ 

F- 16  TEST  RESULTS 

•  DATA  BASE 

-  13  AIRCRAFT 

-  18  MONTHS  (JAN  79  -  JUN  80) 

-  2899  SORTIES 

-  3825  FLYING  HOURS 

•  PART  OF  F-16  OPERATIONAL  SUITABILITY  FQT&E 

•  HILL  AFB  UT 

6-9-3M0 
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TEST  RESULTS 


Figure  3-15 

F- 1 6  ST/BIT  EVENTS 


TEST  RESULTS 
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Figure  3-16  test  results 


F-16  FLIGHT  CONTROL  ST/BIT  PERFORMANCE 
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Figure  3-17 

F-16  MULTIPLEX  BUS  RESULTS 

RESULTS 


TEST  RESULTS 
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Figure  3-18 

EF-1 1 1 A  TEST  RESULTS 

•  DATA  BASE 

-  1  AIRCRAFT 

-  5  MONTHS  (APR-OCT  79) 

-  86  SORTIES 

-  261  FLYING  HOURS 

•  PART  OF  EF-1 11 A  FOT&E 

•  MT  HOME  AFB  ID 


TEST  RESULTS 


Figure  3-19 

EF-1 1 1A  EVENTS 
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Figure  3-20 


EF-1  1 1 A  BIT/BITE  PERr  ORMANCE  TEST  RESULTS 
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Figure  3-21 

WHAT  DO  WE  TELL  USERS? 


CONCLUSIONS  AND  RECOMMENDATIONS 


DEVELOP  DEPLOYMENT/EMPLOYMENT  CONCEPTS 


TRAINING 

SUPPORT  EQUIPMENT 


DEVELOP  REQUIREMENTS  BASED  ON  CONSTRAINTS 

-  MAXIMUM  REPAIR  TIME  ALLOWABLE 

-  MEAN  TIME  TO  REPAIR 

DO  NOT  SPECIFY  FD/F1  PERFORMANCE  IN  TERMS  OF  TRADITIONAL  NUMERIC  VALUES 

INSIST  ON  100%  DIAGNOSTIC  CAPABILITY 

-  MIXTURE  OF  AUTOMATIC  AND  MANUAL  DIAGNOSTICS 


DETERMINE  THE  OPERATIONAL  AND  MAINTENANCE  CONSTRAINTS 

-  DOWN  TIME 

-  SKILL  LEVEL 

-  MANPOWER 


Figure  3-22 

CONCLUSIONS  AND  RECOMMENDATIONS 

WHAT  DO  WE  TELL.  DEVELOPERS? 

•  DEVELOP  FD/F!  SPECIFICATIONS  BASED  ON  USER  REQUIREMENT &  «i:''  CONSTRAINTS 

•  REQUIRE  100%  DIAGNOSTIC  CAPABILITY 

-  MIX  OF  AUTOMATIC  AND  MANUAL  DIAGNOSTIC  CAPABILITY 

-  COMPENSATION  FOR  SHORTFALLS  IN  AUTOMATIC  CAPABILITY 

•  REQUIRE  VERTICAL  TESTABILITY 

•  UNDERSTAND  LIMITS  OF  CURRENT  TECHNOLOGY 

-  REQUIRE  AN  INTEGRATED  DIAGNOSTIC  PLAN 

-  MILESTONES 

-  SUPPORT  OR  DEVELOPMENT  FACILITY 

-  SPECIAL  TEST  INSTRUMENTATION 

-  SYSTEM  DEMONSTRATION  UNDER  OPERATIONAL  CONDITIONS 

-  MATURATION  PROGRAM 

-  CLOSED  LOOP  DATA  SYSTEM 
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Figure  3-23 

CONCLUSIONS  AND  RECOMMENDATIONS 

WHAT  DO  WE  TELL  CONTRACTORS? 

•  INSURE  MANAGERS  CONSIDER: 

-  DIAGNOSTICS  AS  SYSTEMS  ENGINEERING  DISCIPLINE 

-  THE  INTERRELATIONSHIPS  OF  DIAGNOSTICS  TO  OTHER  ILS  ELEMENTS 

-  THE  NEED  TO  USE  DIAGNOSTICS  DURING  DT&E/OT&E 

•  INSURE  DESIGNERS  CONSIDER: 

-  THE  NEED  FOR  PARALLEL  DEVELOPMENT  OF  DIAGNOSTICS  AND  HARDWARE 

-  STRATEGIES  TO  MINIMIZE  THE  OCCURRENCE  OF  FALSE  ALARMS,  CNDs  AND  RTOKs 

-  FAILURE  MODES  EXPERIENCED  IN  THE  OPERATIONAL  ENVIRONMENT 
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Figure  3-24 


CONCLUSIONS  AND  RECOMMENDATIONS 


WHAT  DO  WE  TELL  AIR  STAFF? 

•  RECOGNIZE  THAT  DIAGNOSTIC  CAPABILITY,  NOT  DEGREE  OF  AUTOMATION,  IS  THE  ISSUE 

-  SYSTEM  TRAINING  STILL  REQUIRED 

-  AUTOMATION  TECHNOLOGY  STILL  EVOLVING 

•  DESIGNATE  A  MANAGEMENT  ORGANIZATION  WITHIN  THE  AIR 

-  CENTRAL  REPOSITORY  AND  CORFGRATE  BODY  OF  KNOWLEDGE 

-  REVIEW  AND  PROVIDE  GUIDANCE  ON  DOCUMENTATION  ADDRESSING  DIAGNOSTICS 

-  STANDARDIZE  TERMINOLOGY 

8-9-81-18 


Figure  3-25 

CONCLUSIONS  AND  RECOMMENDATIONS 

WHAT  DO  WE  "ELL  TESTERS? 

•  GET  INVOLVED  EARLY 

-  UNDERSTAND  SYSTEM  DESIGN  AND  CAPABILITIES 

-  PLAN  FOR  SPECIAL  TEST  INSTRUMENTATION 

-  DEVELOP  ANALYTICAL  TOOLS 

-  IDENTIFY  CONTRACTUAL  OATA  REQUIREMENTS 

•  RECOGNIZE  THAT  TRADITIONAL  MAINTAINABILITY  DEMOS  ARE  INADEQUATE 

•  PLAN  FOR  SEQUENTIAL  TESTING  TO  TRACK  SYSTEM  MATURATION 

•  EMPLOY  A  SINGLE  THREAD,  CLOSED  LOOP  DATA  SYSTEM  TO  INSURE  TRACEABILITY 
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figure  3-26 


SUMMARY 

•  100%  DIAGNOSTIC  CAPABILITY  REQUIRED 

-  AUTOMATION  SHORTFALLS  RECOGNIZED  TOO  LATE 

-  SIGNIFICANT  MISSION  IMPACT 

-  PROBLEMS  PERSIST  FOR  LONG  PERIODS 

-  WORKAROUNDS  NOT  ADEQUATE 

-  ORIGINAL  MAINTENANCE  AND  TRAINING  CONCEPTS  INVALIDATED 

•  MANAGEMENT  ATTENTION  AT  THE  HIGHEST  LEVEL  IS  REQUIRED  NOW 

•  DIAGNOSTIC  “CORPORATE  BODY"  ESSENTIAL 
•-81-10 


ADDENDUM  TO  AFTEC  BRIEFING 


Major  Robert  Shafer 
F - 1 6  Suitability  Team 

Major  Bob  Shafer  is  the 
Chief  of  the  F-16  Suita¬ 
bility  ST/BIT  Evaluation 
Team  at  Hill  AFB,  Utah. 

HIGHLIGHTS  OF  THE  PRESENTATION 

The  difference  between  automatic  and  manual  trouble  shoot¬ 
ing  for  the  MUX  BUS  is  not  significant  in  terms  of  MTTR  (2.71 
hours  vs.  2.05  hours)  as  shown  in  Figure  3-27.  However ,  the 
difference  betweeen  automatic  and  manual  trouble  shooting  for 
the  flight  control  system  is  highly  significant  in  terms  of 
MTTR  (11.07  hours  vs.  3-63  hours)  as  shown  in  Figure  3-28.  A 
flight  line  tester  is  required  for  the  flight  control  system 
to  complete  fault  isolation  to  the  failed  component  within  the 
functional  area  identified  by  the  in-flight  BIT.  Addressable 
faults  in  the  flight  control  system  included  166  events. 


DISCUSSION  POINTS 

•  Several  airline  studies  have  indicated  the  presence 
of  chronic  defective  units  such  that  90  percent  of 
the  problems  are  caused  by  10  percent  of  the  units. 
This  experience  was  confirmed  by  Air  Force  work  with 
inertial  navigation  systems. 

•  Airlines  have  found  that  certain  aircraft  with  cabl¬ 
ing  defects  also  contribute  to  high  CND  rates. 

•  The  airlines  are  looking  to  BIT  to  provide  shop  quali¬ 
fication  for  checkout  of  avionics  units  (including 
regulatory  problems). 

•  Carnegie  Mellon  has  done  some  preliminary  work  on 
trend  analysis  concerning  interarrival  rates  of  false 
alarms  indicating  deterioration  of  system  performance 
when  interarrival  times  decrease,  permitting  predic¬ 
tion  of  times  for  removal  of  units,  prior  to  any 
detection  of  such  degradation  by  the  pilot.  Tracking 
by  serial  number  is  required  to  provide  this 
information. 
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•  Tightening  of  the  parameter  ranges  for  fault  detection 
can  increase  false  alarm  rates  excessively. 

•  False  alarms  degrade  confidence  in  the  BIT,  sometimes 
resulting  in  the  flight  crew  ignoring  actual  failures. 

•  Contractors  in  RADC  studies  have  found  that  the 
quality  of  the  data  from  3M  ana  60-1  is  inadequate 
to  perform  the  analyses  required.  A  special  closed- 
loop  monitoring  system  is  required  for  new  complex 
systems . 

•  Ambiguous  fault  isolation  can  result  in  the  replace¬ 
ment  of  multiple  boxes  (some  good,  some  bad)  in  order 
to  restore  the  system.  Some  unit  or  SRA  swapping 
reduces  the  number  returned  for  I-level  repair. 

•  Imposition  of  CND  and  RETOK  requirements  are  dependent 
on  maintenance  philosophy  and  are  difficult  or  impos¬ 
sible  to  impose  on  subcontractors. 

•  Selection  of  faults  for  maintainability  tests  must  be 
randomly  selected  from  a  large  number  of  comprehen¬ 
sively  defined  faults,  including  cable  faults  and 
others  that  are  representative  of  the  operational 
environment  such  as  inputs  from  and  outputs  to  other 
interfacing  equipments. 

«  Support  philosophy  and  training  requirements  are 
developed  based  on  maintainability  predictions;  dif- 
ferencs  between  predictions  and  reality  can  cause 
significant  impacts  in  support  and  training. 

•  The  airlines  are  accumulating  data  bases  of  character¬ 
istics  of  reliabilities  demonstrated  and  fault  isola¬ 
tion  capabilities  on  different  types  of  units  (LRU's, 
circuit  boards,  etc.)  in  order  to  determine  realistic 
characteristics  to  expect  for  these  items  in  the 
future.  In  digital  board  testing,  the  isolation  capa¬ 
bility  has  been  around  43  percent  vs.  90  percent 
claimed . 

o  Some  of  the  larger  airlines  do  not  buy  diagnostic 
software  from  the  vendors  but  build  it  themselves 
after  delivery  when  the  hardware  is  completed. 

•  The  airlines  use  A&P  personnel  for  flight  line 
maintenance,  reserving  more  highly  qualified  personnel 
for  shop  maintenance. 

•  The  trend  as  shown  in  the  E-3A  is  from  the  "smart 
machine,  dumb  man"  to  the  "smart  machine,  smart  man" 
concept  to  cover  the  areas  where  BIT  fails  to  detect 
and  isolate  the  problem. 
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F- 1 6  SELF  TEST/BIT  IMPLEMENTATION 
AND 

LESSONS  LEARNED 
Gordon  England 

GENERAL  DYNAMICS,  FORT  WORTH  DIVISION 

Mr.  England  is  the  Director 
of  Avionics  Systems  for  the 
Fort  Worth  Division  of 
General  Dynamics 

HIGHLIGHTS  OF  THE  PRESENTATION 

The  presentation  covers  four  specific  areas:  P-16  avionic 
descriptions  and  ST/BIT  requirements;  F-16  ST/BIT  design  ap¬ 
proach  and  application  of  previous  lessons  learned;  F-16  ST/BIT 
results;  and  application  to  future  programs. 

F-16  capabilities  are  shown  in  Figure  4-1;  F-16  ST/BIT 
requirements  are  shown  in  4-2.  General  overall  requirements 
are  specified  for  the  system  and  then  specific  subsystem  re¬ 
quirements  are  specified.  Since  the  flight  control  computer 
is  quad  redundant  and  monitors  all  failures,  no  quantitative 
ST/BIT  requirement  was  stated. 

Requirements  imposed  by  General  Dynamics  on  suppliers  are 
shown  in  Figure  4-3;  they  are  seen  to  be  "stiffer"  than  those 
imposed  by  the  Air  Force  in  terms  of  detection  and  isolating 
to  FLUs.  In  addition,  General  Dynamics  has  imposed  additional 
requirements  of  isolating  to  failed  function?  within  FLUs, 
which  has  resulted  in  about  400  different  faults  being  reported 
rather  than  just  the  total  number  of  LRUs.  This  secondary 
fault  reporting  was  imposed  by  GD  on  the  suppliers.  Identifi¬ 
cation  of  functional  failures  permits  the  pilot  to  evaluate 
the  effect  of  a  failure  on  the  weapon  system. 

Figure  4-4  shows  the  approach  taken  by  GD  to  establish  a 
total  integrated  ST/BIT  program  for  the  F-l6.  This  program  was 
based  on  inputs  from  a  number  of  agencies  such  as  The  RAND 
Corporation  and  AFTEC,  and  based  on  a  number  of  earlier  systems 


such  as  the  P-15  and  the  F-lll.  Each  item  of  the  program  will 
be  discussed  in  detail.  All  items  of  the  F-16  ST/BIT  design 
approach  are  considered  essential  to  a  successful  ST/BIT 
program. 

It  is  important  to  state  at  this  point  that  it  is  essen¬ 
tial  that  one  individual  be  assigned  ST/BIT  responsibility. 

This  assures  a  completely  coordinated  fault  detection  and 
isolation  development  resulting  in  a  good  operational  capa¬ 
bility.  Also  a  single  point  responsibility  will  result  in  a 
well  organized/well  managed  effort  to  meet  established  goals. 

The  most  deliberate  front  end  item  requirement  is  parti¬ 
tioning  the  system  properly  to  incorporate  ST/BIT,  as  shown  in 
Figures  *1-5  and  *1-6.  The  philosophy  is  that  subsystems  are 
self-contained  and  perform  total  functions,  outputting  whole 
values  (not  delta  values)  under  the  overall  control  of  the  FCC. 
The  FCC  is  a  32  K  word  machine,  of  which  27  K  words  are  pres¬ 
ently  used.  Simple  interfacing  was  also  considered  essential, 
as  indicated  in  Figure  *1-7.  For  example,  the  (approximately) 

70  wires  between  the  radar  and  display  in  the  F-lll  were 
reduced  to  11  in  the  F-16.  One  universal  multiplex  data  line 
was  used  for  system  communication.  All  symbology  is  generated 
within  the  display  based  on  commands  from  the  FCC. 

A  stable  configuration  was  considered  essential  and  estab¬ 
lished  as  shown  in  Figure  *1-8;  primarily  because  numerous 
changes  can  impact  the  effectiveness  of  ST/BIT  and  partly 
because  changes  required  coordination  with  all  EPG  countries. 
This  stability  allowed  ST/BIT  to  be  incorporated  in  block 
changes . 

ST/BIT  was  integrated  into  subsystems  as  shown  in  Figure 
*1-9.  No  printers  or  hardcopy  readouts  are  provided  so  as  to 
not  introduce  another  item  subject  to  failure.  For  post  flight 
debriefing  and  maintenance,  the  operator  writes  down  the  alpha¬ 
numeric  code  describing  the  failure  from  the  FC  Navigational 
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Panel.  Most  of  the  systems  are  operating  continuously  and  are 
continually  monitoring  themselves,  minimizing  the  amount  of 
interruptive  BIT  required. 

Early  testing  was  utilized  as  shown  in  Figure  4-10.  All 
ST/BIT  failure  indications  were  tracked  for  accuracy  of  report¬ 
ing  during  laboratory  and  flight  tests.  Any  ST/BIT  anomolies 
were  corrected  during  the  development  phase.  The  avionics 
equipment  bay  includes  a  nose  section  of  the  aircraft  and  is 
tied  to  a  software  development  facility  (dynamic  test  station) 
to  provide  an  integrated  system  to  permit  formal  testing  before 
demonstration  in  the  aircraft.  Algorithm  deficiencies  (e.g., 
negative  velocities  in  the  INS  on  the  ramp)  have  been  detected 
by  this  method,  eliminating  possible  numerous  CNDs  on  the  NAV 
Computer.  Tech  orders  are  oriented  toward  field  usage,  includ¬ 
ing  using  T.O.  writers  in  actual  maintenance  as  shown  in 
Figure  4-11. 

ST/BIT  failures  are  reported  as  to  function  failed,  time 
of  failure,  and  intermittency  as  shown  in  Figure  4-12.  At  the 
present  time,  this  information  is  not  fully  utilized  by  the 
AIS,  since  procedures  are  not  adapted  to  its  use.  Takeoff  and 
landing  times  are  also  recorded.  Proper  use  of  these  data 
(second  tier)  in  the  AIS  could  likely  reduce  initial  test  time 
by  50  percent  or  more.  The  AIS  development  lagged  the  avionics 
in  order  to  let  it  mature,  but  did  not  take  advantage  of 
secondary  fault  data. 

Data  and  test  requirements  were  integrated  into  the  de¬ 
sign  process  as  shown  in  Figure  4-13;  verification  is  required 
throughout  the  program,  using  ST/BIT  as  shown  in  Figure  4-14. 

ST/BIT  results  at  the  subcontractor  level  are  shown  in 
Figure  4-15.  The  same  induced  faults  were  also  tested  with 
the  AIS.  Everything  reasonable  in  ST/BIT  was  done  to  achieve 
95  percent  FD/FI  without  doing  anything  ridiculous  to  achieve 
a  specific  percentage.  An  estimate  of  10  percent  additional 
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cost  for  ST/BIT  appears  reasonable  for  acquisition  costs.  The 
difficulties  in  early  measurement  of  ST/BIT  field  data  are 
shown  in  Figure  4-16.  The  data  base  consisted  of  an  l8-month 
time  frame,  22  aircraft,  2899  sorties/3816  flight  hours,  and 
2918  write-ups.  ST/BIT  results  in  the  field  are  shown  in 
Figures  4-17  and  4 — 18 .  (The  total  of  the  "engineering  defi¬ 
ciencies"  and  "no  trials"  in  Figure  4-17  corresponds  to  the 
"special  category"  data  of  the  RADC  presentation.)  CND  rates 
shown  in  Figure  4-17  are  misleading  since  they  do  not  consider 
high  reliability  items.  However,  CNDs  reduce  user  confidence. 
The  Suitability  Team  (Figure  4-19)  is  useful  in  determining 
the  specific  action  required  to  correct  the  deficiencies  and 
mature  the  BIT. 

ST/BIT  general  lessons  learned  are  shown  in  Figures  4-20. 
Specification  requirements  recommended  are  shown  in  Figure  4- 
21.  False  alarms  should  be  eliminated  and  not  permitted  in 
the  specif icacion ;  threshold  and  anomoly  duration  criteria 
should  be  set  such  that  momentary  faults  due  to  power  trans¬ 
ients,  for  example,  are  not  recorded  as  faults  but  at  the  same 
time  record  real  faults  that  affect  system  performance.  CNDs 
and  RETOKs  should  also  be  eliminated  ultimately. 

The  advantages  of  implementing  BIT  at  the  subsystem  level 
are  shown  in  Figure  4-22;  details  of  the  type  of  data  readouts 
are  shown  in  Figure  4-23-  The  importance  of  emphasizing 
management  and  control  and  of  incorporating  ST/BIT  as  a  first 
line  requirement  early  in  the  design  of  a  system  are  shown  in 
Figure  4-24.  Demonstration  and  test  requirements  are  shown 
in  Figure  4-25.  Intermittents  can  be  addressed  by  use  of 
storage  and  history  provisions  as  indicated  in  Figure  4-26. 
Recommendations  for  field  support  and  evaluation  are  shown  in 
Figure  4-27.  T.O.s  should  be  prepared  by  personnel  with 
"hands-on"  experience  who  participate  in  the  initial  system 
Integration  activities  and  continue  through  flight  test  on  the 
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development  hardware  to  ensure  maximum  knowledge  of  systems, 
features,  handling  peculiarities  and  general  eccentricities. 
Training  simulators  that  use  the  real  hardware  can  now  be 
implemented  owing  to  extensive  digital  systems  that  are  soft¬ 
ware  intensive,  as  shown  in  Figure  4-28.  BIT  can  be  correlated 
with  fault  isolation  at  other  maintenance  levels  as  shown  in 
Figure  4-29.  The  design  lessons  for  subsystem  BIT  are  given 
in  Figure  4-30. 

A  summary  of  the  F-l6  experience  is  given  in  Figure  4-31. 
The  F-16  has  shown  the  highest  sortie  rate  in  the  Air  Force, 
used  less  spares  than  projected,  and  achieved  both  in  a 
relatively  short  period  of  time. 

DISCUSSION  POINTS 

•  A  development  program  requires  a  test  and  evaluation 
effort  similar  to  that  performed  by  AFTEC  on  the  F-16 
in  order  to  be  successful.  This  is  one  of  the  most 
important  lessons  learned  from  the  F-16. 

•  The  question  is  whether  a  program  such  as  the  AFTEC 
F-16  test  program,  is  affordable  on  all  programs; 
this  is  in  view  of  the  fact  that  the  Air  Force  has 
approximately  90  major  programs  and  200  minor  pro¬ 
grams  in  progress  at  the  present  time.  However,  it 
was  pointed  out  that  the  F-16  Suitability  Team  in¬ 
volved  only  a  few  people  in  the  field  whose  inputs 
were  fed  directly  to  the  GD  engineers  at  Fort  Worth, 
permitting  rapid  corrective  action.  Consequently, 
considerable  benefits  were  achieved  with  relatively 
low  up-front  investment.  The  F-16  Suitability  Team 
plans  to  recommend  to  TAC  that  the  activities  of  the 
Team  be  continued  at  the  present  level  rather  than 
be  curtailed  (as  is  currently  planned). 

•  Externally  generated  BIT  is  more  difficult  to  imple¬ 
ment  than  ST  conducted  within  each  subsystem  partly 
because  of  the  numerous  Class-2  changes.  The  inertial 
system  on  the  F-lll  had  13,000  Class-2  changes. 

«  The  inertial  navigacion  system  includes  10  percent  of 
its  piece  parts  for  self  test;  however,  because  of  ST- 
BIT,  95  percent  FD/FI  is  achieved  at  reasonable  cost. 

«  ST  is  incorporated  in  sensor  encoders  (RIUs)  in  the 
F-16.  This  minimized  fault  isolation  procedures. 
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•  GFE  sometimes  presents  a  problem  in  terms  of  testing 
and  reporting  its  own  condition  and  the  condition  of 
its  sensors. 

•  Design  problems  may  manifest  themselves  as  a  number  of 
different  faults.  High  skill  engineers  are  required 
to  identify  problems  such  as  these. 

•  Fault-tolerant  systems  have  been  investigated  by 
General  Dynamics  in  an  IR&D  program  (which  is  about 
to  become  a  government-funded  program).  It  includes 
small,  standard  throw-away  modules  with  no  I-level 
maintenance,  implementing  the  entire  avionics  system 
with  eight  standard  modules. 

•  Access  to  faults  stored  in  non-volatile  memory  should 
be  achieved  at  0-level  (preferably  without  powering 
up  the  aircraft)  and  at  I-level. 

•  Maintainability  demonstration  requires  stabilized 
hardware  and  software  and  cannot  be  moved  to  an  early 
phase  in  the  program. 

•  RIWs  were  used  with  suppliers  to  provide  incentives 
for  achieving  reliability  of  their  subsystems. 

•  FEMAs  are  a  design  tool,  providing  a  discipline  for 
ST/BIT. 

•  No  formal  "sneak  circuit"  analyses  were  performed. 

It  is  of  questionable  testability  value  because  of 
the  dynamic  nature  of  the  systems. 

•  It  is  estimated  that  two  to  three  years  are  required 
in  the  field  in  order  to  mature  the  ST/BIT.  Part  of 
this  time  is  related  to  the  quality  of  the  design, 
the  suitability  of  the  failure  data,  and  the  ECP 
process.  Changes  were  approximately  75  percent  <-oft- 
ware  and  25  percent  hardware.  Software  changes  were 
sometimes  used  to  correct  hardware  deficiencies. 

•  Some  airlines  are  now  specifying  MTBRs  in  order  to 
address  the  CND  problem. 

•  The  airlines  are  also  finding  that  two  years  are 
required  to  mature  new  avionics  systems. 

•  Nuisance  warnings  are  a  problem  with  the  airlines, 
especially  in  ground  proximity  warning  systems. 

•  There  was  a  significant  level  of  discussion  on  the 
issue  of  where  there  is  such  a  characteristic  as  a 
"false  alarm"  and  where  a  "false  alarm  rate"  should 
be  included  in  specifications.  On  one  hand  it  was 
argued  test  any  time  there  is  an  indication  of  failure 
to  the  operator,  there  is  a  need  to  correct  some 
aspect  of  either  the  system  or  the  BIT  logic,  and 
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therefore  there  is  no  such  thing  as  a  false  alarm. 

On  the  other  hand,  it  was  argued  there  are  certain 
transients  that  occur  which  cause  failure  indications 
but  which  cannot  be  completely  eliminated.  Therefore 
some  minimum  false  alarm  level  has  to  be  specified. 

A  consensus  was  not  reached. 
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Figure  4-1 


F-16  AVIONIC  SYSTEM  CAPABILITIES 


AIR-TO  AIR 

'SM. 


•  Accurate  gunnery 

•  Look  down  into  clutter 

•  Automatic  radar  acquisition 

•  Single  switch  entry  into  air-to-air 

•  Sidewinder  dynamic  launch  2one 
computation 

•  Provision  for  radar  missile 

v _ _ _ 


AIRTOGROUND 


SINGLE  POINT  MODE 
&  WEAPON  CONTROL 

•  Accurate  weapon  delivery 
J  VISUAL 
J  BLIND 

•  Offset  and  beacon  radar  bombing 

•  High  resolution  ground  map 

•  E/O  weapon  delivery  (Hobo  &  Maverick) 

•  Provision  for  laser  spot  receiver 


••  7  ?  < 


Mi 

;  :  VC 

l| 

oh uo !  I 

Cl 

i  * 

£ 

•oo  n  ] 

f 

•o|  ; 

r  OPERATIONS  &  SURVIVABILITY 

vground  IFF,  TACAN,  ILS  and  inertial  navigation 
hspenser  and  ECM  pod  controls 

•  Standard  communications,  air-tc 

*  Threat  warning,  chaff  and  flare  < 

Figure  4-2 

F-16  SELF  TEST/BUILT-IN-TEST  (ST/BIT)  REQUIREMENTS 


•CENERAL  REQUIREMENTS  16PS001  SYSTEM  SEPCIFICATION 

•  REMOVE  AND  REPLACE  AIRCRAFT  EQUIPMENT  WITHOUT  NEED  FOR  ADJUSTMENT  EXCEPT 
AS  PROVIOEO  BY  BIT  CAPABILITY 

•  MINIMIZE  REQUIREMENT  FOR  FLIGHT  LINE  SE  FOR  AVIONICS 

•  MAXIMIZE  USE  UF  ST/BIT  FOR  AVIONICS  SYSTEM  CHECKOUT  AND  FAULT  ISOLATION 


•  SPECIFIC  REQUIREMENTS  16PS002  AIR  VEHICLE  SPECIFICATION 

•FIRE  CONTROL  RADAR  -  DETECT  ANO  ISOLATE  TO  LRU  FOR  95%  OF  MALFUNCTIONS 
FALSE  ALARM  (FA)  <1% 

•  HUO  SET  OETECT  CERTAIN  FAILURES  'f  suit  AND  90%  OF  FAILURES  IN  SYMBOL 
GENERATOR  <I%FA 

•  FIRE  CONTROL  TUMPUTER  -  Rt'K'  REPORT  95%  OF  MALFUNCTIONS 

•  E/O  OISPLAY  -  DETECT  CERTAIN  ?'•<•  .RES  IN  OISPLAY  AND  95%  OF  FAILURES  IN  SG/EU 
<;1%FA 

•  INERTIAL  NAV  SET  -  OETECT  95%  OF  FAILURES.  ISOLATE  95%  OF  DETECTEO  FAILURES 

«.  1%  FA 


FLIGHT  CONTROL  COMPUTER  -  INTEGRATED  OPERATIONAL  CHECKOUT  ASPART  OF 
SYSTEM  DESIGN  NO  QUANTITATIVE  ST/BIT  REQUIREMENT 


Figure  4-3 

SUBCONTRACTOR  ORGANIZATIONAL  LEVEL  TEST  REQUJREMENTS 


SUBCONTRACTOR 

ITEM 

- - — - - - 

REQUIREMENT 

WESTINGHOUSE 

Fue  Control  R*.Eiar 

95%  Failure  Detection 

95%  Isolation  to  FLU 

MARCONI  ELLIOTT 

HUD  Set 

95%  Failure  Detection 

95%  Isolation  to  FLU 

OELCO 

Fire  Control  Computer 

95%  Failure  Detection 

95%  Isolation  to  FLU 

GO 

SMS 

95%  Failure  Detection 

95%  Isolation  to  FLU 

KAISER 

E/0  Display  Set 

95%  Failure  Detection 

95%  Isolation  to  FLU 

SINGER  KEARFOTT 

Inertial  Nav  Set 

95%  Failure  Detection 

95%  Isolation  to  FLU 

GD/LEAR  SIEGLER 

1 - - 

Flight  Control 

Detailed  Sell  Test  Design  Requirements 
Specified  in  Vendor  Spec  and  in  Teaming 
Arrangement.  Quad  Redundant  System 
Monitors  All  Abnormalties 

figure  4-’! 

F-16  ST/BIT  DESIGN  APPROACH  &  PREVIOUS  LESSONS  APPLIED 


•  A  SYSTEMS  APPROACH 

.A  MATURE  EQUIPMENT  SUITE 

•  SIMPLE  INTERFACES 

•  STABLE  CONFIGURATION 

.  BIT  INTEGRATED  INTO  SUBSYSTEM  DESIGN 
.  EARLY  TESTING  OF  DETECTION  ANO  ISOLATION  CAPABILITY 

•  ADEQUATE  FAULT  ISOLATION  FUOCEOUOES  IT  0.,.  TOO  «I«CRE»  A«0  MAIUTENAUEE 
PERSONNEL 

.  ADDED  FAULT  CATA  FOR  SHOP  PERSONNEL 

•  MANAGEMENT  EMPHASIS 

•  MEANINGFUL  DATA  ANO  TEST  REQUIREMEHIS 

•  VERIFICATION  TESTING 

•  FIELD  SUITABILITY  TEAM 
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4-5 


F-16  AVIONIC  SYSTEM  PARTITIONED  FOR  MAINTENANCE 


STORES  UAMACEUEN I  SET  KIU) 


PROGRAMMABLE  CENTRAL  INTERFACE  UNIT 

♦  Ow**  R«u«<.u«» 

•  U«ne<f  Rn«i< 


flM  COKIROl  CONTOUR 


SYSTEM  f  UNCTIONS  IN 

fire  control  computer 

«  f'ro^anwntbk  Co<» 

•  AWmcny  A  R«i4*»« 

•  Co*"W«t«  (UmMtlKMd  0<Xur*v«oUI>onl 

•  HOL  l<x  (mol  M*<o[k\kk» 


INERTIAL  NAVIGAIICN  UNIT 


SENSORS  TROVIOE  TOTAL  SENSOR  FUNCTION 

•  Hm  IP>««  OwnTi(KttM<i 

•  C—i  I*uiJo» 

•  SmmAIkU  md  Fndt 

OtiKiKxi  *nd  l*oUi-oo 

•  0**<kx>n>«nt 

•  A(M<nf  N«w  C*oxtxl«>*i 


0UU510^iiJ  HUO  ANO  RAOAR/f  0  OtSfl  AYSHAVE  SETA  RATE 

'FIRMWARE"  JYUIOL  GENERATORS 
|W*  •  CMytMtWI  5  !•»♦«>♦*»  N<tnlH< 

*  Syatk«R  C*«l>»A*d  Ay  f  CC 


•  AIR  DATA  CQMPUTtn 

•  growth  systems 


Figure  4-6 

DESIGN  SIMPLICITY  AND  MATURITY 


RADAR/EO  DISPLAY 


FIRE  CONTROL  COMPUTER 


HEAD  UP  DISPLAY 


•  TV  DISPLAY 

•  SIMPLE  RADAR  INTERFACE 

•  1500  PARTS 


[LESS  THAN  21,000  PARTS  IN  TOTAL  FIRE  CONTROL  SYSTEM 
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Figure  *1-7 


SIMPLE  INTERFACES 

-  SYSTEM  STRUCTURE  RESULTS  IN  SIMPLIFIED  INTERFACE 

•  DISTRIBUTED  PROCESSORS  ELIMINATE  COMPLEX  CROSSTALK 

•  SCAN  CONVERTER  IN  RADAR  REDUCES  INTERFACE  BETWEEN 
RAOAR  AND  DISPLAY  (Only  11  Wires  in  F-16) 

•ONE  BOX  INU  RESULTS  IN  CAL  DATA,  FAILURE  DATA  AND 
MOST  INTERFACES  ALL  WITHIN  ONE  FLU 

•SYMBOLOGY  GENERATION  WITHIN  EO  DISPLAY  AND  WITH¬ 
IN  HUD;  HENCE,  NO  CONSTRUCTION  SIGNALS  IN  INTERFACES 

•  Ml L-STD-1 553  MUX  RESULTS  IN  ONE  UNIVERSAL  DATA  LINE 

Figure  *1-3 

REASONABLY  STABLE  CONFIGURATION  ESSENTIAL 

•  PRIOR  PROGRAMS  SUFFERED  FROM  NUMEROUS  CHANGES  TO  THE 
AVIONIC  SYSTEMS 

•  CHANGES  TO  BOXES  IMPACTED  BIT  AND  INVALIDATED  PRIOR 
M-OEMO  RESULTS 

•INADEQUATE  FIELD  PROCEDURES  RESULTED  FROM  TIME  LAGS 
IN  INCORPORATING  CHANGE  RELATED  DATA 

•MAINTENANCE  PERSONNEL  EXPERIENCED  NEGATIVE  LEARNING 
CURVES 

•  CHANGES  DEGRADED  RELIABILITY 

•  BOTH  GD  AND  THE  GOVERNMENT  ON  THE  F-16  WERE  COMMITTED 
TO  KEEP  CHANGES  TO  A  MINIMUM 

•  EPG  PARTICIPATING  REDUCED  CHANGE  LEVEL 

•  BLOCK  POINTS  FOR  CHANGE  INCORPORATION 

*19 
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Figure  4-9 


BIT  INTEGRATED  INTO  SUBSYSTEM  DESIGN 


*  Fault  Detection  and  Isolation 


Implemented  in  Each  Subsystem 


FIRE  CONTROL  FIRE  CONTROL 

NAVIGATION  PANEL  COMPUTER  (FCC) 


Failure  Data 
Repotted  to  FCC 


STORES  MANAGEMENT 


♦  Fault  Readout 

*  Pilot  or  Maintenance 
Crew  Initiate  Fault 
Isolation  Routines 


•  Discriminates 
Between  Pilot 
and  Maintenance 
Data 

•  Stores  Data  lor 
Display 


•  SUPPLIER  OOES  HIS  COMPLETE  JOB 

•  SIMPLE  INTERFACES  REOUCE  FAUlT  ISOLATION  PROBLEM 

•  SUPPLIER  HAS  HAHDWARE/SOFTWARE  DESIGN  FLEXIBILITY 


HEAO  UP  OISPLAY 


OTHER 

SUBSYSTEMS 

Figure  4-10 


EARLY  TESTING 


I'KI  VKHIS  I'KOIII  1  \l  - 

•  BDiLT  IN-TEST  NOT  TESTED  AND  VERIFIED  EARLY  IN  DEVELOPMENT 

•  Performance  Testing  Had  Priority 

•  With  Central  Processor  Concept  Subsystem  Test  of  Bui\  In  Test  Limited 

•  System  evel  Test  Was  too  Little  and  too  Late 

RESUL  T  -  Atjny  Self  Test  end  Built  in  Test  Problems  Fust  Appeared  j! ter  Deployment 


I  Id  UTUO  \(  II 


•  TEST  EARLY 

•  PARTITIONED  DESIGN  PERMITS  TEST  AT  LOW  LEVEL  AND  IN  DISCRETE  BLOCKS 

•  More  Testing  During  M-Oemo  at  Box  Level 

e  Test  Built  in  Test  During  Box  Level  Acceptance,  Qual,  Reliability  and  Maintainability 
Testing 

•  SYSTEM  TEST  DURING  INTEGRATION  TESTING  IN  AVIONICS  EQUIPMENT  BAY 

•  AIRPLANE  TEST  OURING  AIRPLANE  No  3  FLIGHT  TEST 

RESUL  f  -  Early  Detection  o!  Problems  with  Time  to  Fix  before  Deployment 


& 
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Figure  *1-11 


NEWT.O.  DEVELOPMENT  APPROACH 


1 

pin  vious  i*i(oit i.i  m 

n 

1 

•  ENGINEERING  DEVELOPED  PROCEDURES  WERE  FACTORY  RATHER  THAN  FIELD 
ORIENTED 

•  TECH  WRITERS  PARTICIPATED  AFTER  INTEGRATION  AND  LAB  TEST  COMPLETE 

•  LIMITED  "IN-SERVICE"  USE  TO  VALIDATE  TECH  ORDERS  BEFORE  DEPLOYMENT 

RESULT  -  Inadequate  Procedures  in  the  Field 

-  Equipment  Removals  and  Retest  OK's 


I  l<>  U’I’UO  \(  II 


•  USE  T.O.s  IN  THE  PLANT  -  NOT  MANUFACTURING  TEST  PROCEDURES 
x  Engineering  and  Tech  Writers  to  Jointly  Develop  T.O.s 
xT.O  s  to  Be  Used  During  Lab  integration  and  Early  Flight  Test 
^Organization  Level  T.O  s  to  Be  Released  on  Airplane  No  3  for  Validation 

RESULT  -  Usable  T  Os  for  USA  F  Operational  Support 


Figure  4-12 

DESIGN  SENSITIVE  TO  AIS  NEEDS 

m  VIOUS  I’HOHI-I.M  | - — 


•  OATA  REPORTED  TO  CENTRAL  COMPUTER  LIMITED  TO  GO  NO  GO 
v  Shop  Personnel  Had  No  Data  Relevant  to  Failure  Mode 
x  Intermittent  Failures  Not  Captured 

RESULT  -  Numerous  Retest  OK's 

Difficult  Fault  Isolation  in  Shot > 


1  l(i  Xl'PKOU  II 


•  INTERNAL  SUBSYSTEM  FAULT  DATA  IS  REPORTED  TO  FCC 
x  Failure  Functions  Within  Each  FLU  Are  Reported 
x  Failures  Tagged  by  Mission  Phase  anil  Tune 

RESUL  T  -  Shot)  Has  Detail  Failure  Data  to  Hero  in  on  Sad  Part 
Odds  Improved  in  Finding  Inter  mi  I  tents 


51 


*x  >  «  £  '■*  * 


Figure  *J-i3 


ST/BIT  AN  INTEGRAL  PART  OF  SYSTEM  AND  SUBSYSTEM  TESTS 

•  SUBCONTRACTOR  -  SUBSYSTEM  LEVEL 

•  Module  Build  Up  Testing 

•  LRU  Testing,  Engr  Evaluation,  Qual,  M-Demo 

•  Formal  Acceptance  Testing 

•  CONTRACTOR  -  SYSTEM  LEVEL 

•  Hardware  Integration  -  with  Other  Avionic  Systems 

•  Software  Development  -  Operational  System  Mode  Development 

•  Avionics  Equipment  8ay  -  Flight  Simula!  ion 

•  FSD  Flight  Test 
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Figure  *1-15 


SUBCONTRACTOR  ORGANIZATIONAL  LEVEL  TEST  RESULTS 


SUBCONTRACTOR 

ITEM 

GEN  DYNAMICS 
REQUIREMENT 

SUBCONTRACTOR 

INITIAL 

PREDICTION 

RESULT 

Weslingliouse 

Fite  Control  Radar 

95%  Failure  Detection 

>  95V. 

94%  Failure  Detection 

95%  1  solatia:)  lo  FLU 

>  97V. 

95%  Isolation  to  FLU 

Maicoiu  Elliott 

MUD  Set 

95%  Failure  Detection 

>  95  V. 

94%  Failure  Detection* 

95%  Isolation  to  FLU 

>  95V. 

95%  Isolation  to  FLU 

Oelco 

fne  Control  Computer 

95%  Failure  Detection 

98V. 

95%  Failure  Detection 

95%  Isolation  to  FLU 

98V. 

95%  Isolation  to  FLU 

GO 

SMS 

95%  Failure  Detection 

In  Work 

95%  Failure  Detection 

95%  Isolation  lo  FLU 

95V.  hoUtioo  lo  FLO 

Kanti 

E/0  Oiiplay  Set 

95%  Failure  Detectiun 

95V. 

94%  Failure  Detection* 

95%  Isolation  to  FLU 

95V. 

95%  Isolation  to  FLU 

Smgei  Keailott 

Inertial  Nav  Sel 

95%  Failure  Detection 

>95V. 

95%  Failure  Detection 

95%  Isolation  to  FLU 

>  95V. 

95V.  Rotation  lo  FLU 

GO/Leai  Sieglei 

Flight  Control 

•Detailed  Sell-Test  Oe 

NA 

95%  Failure  Detection 

Sign  Reqmt’s  Specified 

95%  Isolation  to  FLU 

in  Vendor  Spec  and  in 

Teaming  Arrangement. 

*Bit  (Visual)  Used  to 

Quad  Redundant  Sysien 

Supplement  ST  to 

MomKm  All  Atinoimillies 

1 

Achieve 94*  Detection 

Figure  4-16 

EARLY  MEASUREMENT  OF  FIELD/OPERATIONAL  LEVEL  IS 
VERY  DIFFICULT 


•  ONE  on  TWO  SUBTLE  DESIGN  PROBLEMS  CAN  APPEAR  TO  BE  GROSS 
ST/UIT  PROBLEMS 

*  RAW  DATA  NOT  RELEVANT 


•  CAN'T  SEPARATE  INTERMITTENTS  AND  CNDs  FROM  DATA  BASE 


•  EARLY  INEXPERIENCE/IMMATURITY  OF  PERSONNEL  AND 
SUPPORT  EQUIPMENT  APPEAR  ASST/BIT  PROBLEMS 


COMPREHENSIVE  DATA  NEEDED  WITH  ON  SITE 
SKILLED  ENGINEERS  TO  DRAW  VAl  ID  CONCLUSIONS 
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Figure  ^-17 


DETAILED  UNDERSTANDING  OF  DATA  YIELD  DIFFERENT  RESULTS 


(CNO  EXAMPLE) 


SUBSYSTEM 

HAW  ST/BIT 
CNO  HATE 

SOURCE  OF  CNO 

ACTUAL 

ST/BIT 

CNO  HATE 

FIHE  CONTROL 
COMPUTER 

65% 

62%  Confirmed  as  Several  FCC  OFP  Problems  Since  Correcied 

3%  0  Unit)  Instil lictent  Data  to  Analyte 

3% 

HEAD  UP 
DISPLAY  SET 

79% 

64%  Due  to  Only  Two  Idenlilied  Sell  Test  Design  Inconsistences 
13%  Due  to  1  Intermittent  EU  (HarJwaie  Failure) 

6%  Due  to  Ollier  A/C  System 

7% 

IIAOAU/EO 

IMSPI  AY  SET 

67% 

30%  Due  to  Unique  MOT&E  Class  II  Muds 

20%  Due  to  MF  Is  001  &  008  Under  Active  Investigation 

1 7%  One  T  line  Occunences  (July  79  -  Jan  60)  on  6  Dillerem 

A/C  and  Have  Not  Hcpealed 

o  i ;% 

INERTIAL 

NAVIGATION 

SLI 

22% 

14%  Due  to  Recycling  ol  INS  Power  in  Less  than  1  Minute  - 
Violates  Proceduie 

6%  Intermittent  Failing  Ha.  !waie  Likely 

1%  Coulmned  Iniennithnt  Faituie 

I  6% 

EIRE  CONTROL 
RADAR 

34% 

4%  S/T  Timing  Problem  Fix  in  Latei  Contig 

0%  Turn  On  Bit  Problem 

2%  Test  Mech  Problem  -  Fix  in  Latei  Conlig 

4%  S /(  A/D  Circuit  Checks  fixed  in  Later  Conlig 

10%  Unknown 

1!»% 

Willi  1)1  I  AILED  UNDERSTANDING  Of  OA  1A  A  PltOI'Ell  PEHSI’ECl  IVE  AND  APPHOPI1IA  1 E 
CORRECTIVE  ACTION  CAN  HE  TAKEN 


Figure  4-19 

SUITABILITY  TEAM 

•  FLIGHT  TEST 

•  TRACK,  ANALYZE  AND  TAKE  CORRECTIVE  ACTION  ON  EVERY  SELF  TEST/ 

BIT  REPORT 

•  F-16  DEPLOYMENT 

•MANAGEABLE  SAMPLE  SIZE  FORTRACKING  AND  DETAILED  FOLLOW  THROUGI 

•  SUITABILITY  TEAM 

•Contractor  and  USAF 

•  Complete  and  Accurate  Data  Base 

•  On-Site  Follow  Through 

•  Common  Data  Base 

•  "Qual"  Type  Accelerated  Data  (MOT&E) 

•  SUPPLEMENT  SUITABILITY  TEAM  OATA  WITH  AIS  AND  DEPOT  CND/RTOK 
MONTHLY  REPORTS 

Figure  4-20 

GENERAL  SYSTEM  LESSONS 

•  MAXIMIZE  NON-INTERRUPTIVE  TEST 

•  SIMPLIFIES  SYSTEM  OPERATION 

•  MINIMIZES  OPERATOR  INTERACTION 

•  STORE  AND  REPORT  ALL  FAILURE  DATA 

•  MAY  PERMIT  SRU  IDENTIFICATION  OIRECTLY 

•  KEEP  VISUAL  FAULT  DETECTION  REQUIREMENTS  TO  A  MINIMUM 

•SUBSYSTEM  READY  DISCRETE 

•  SUBSYSTEM  HAS  POWER  APPLIED 

•  ENOUGH  TIME  ELAPSED  FOR  ALL  BITS  AT  THE  MUX  INTERFACE 
TO  BE  CREDIBLE 
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Figure  4-21 


SPECIFICATIONS 


•  BIT  FAULT  DETECTION  AND  ISOLATION  REQUIREMENTS  ARE 
REASONABLE  AND  ACHIEVABLE  IN  THE  95%  RANGE 


•  FALSE  ALARM  REQUIREMENTS  RELATIVE  TO  BIT  SHOULD  BE 
CHANGED  FROM  1%  TO  NO  FALSE  ALARMS  -  NOT  MEANINGFUL 
AND  CAN'T  BE  MEASURED 

NOTE  Intermittent  Failures  Are  Not  Considered  False  Alarms 


•  MAINTAINABILITY  DEMONSTRATIONS,  QUAL  TESTS  AND 
ACCEPTANCE  TESTS  SHOULD  BE  TIED  TO  BIT  REQUIREMENTS; 
i.e.,  BIT  SHOULD  BE  USED  AS  A  PRIMARY  INDICATION  OF 
FAILURES  DURING  DEMONSTRATIONS 

Figure  4-22 

SYSTEM-VS-SUBSYSTEM  BIT 

•  8IT  FOR  A  GIVEN  SUBSYSTEM  SHOULD  BE  TOTALLY  SELF  CONTAINED 
WITHIN  THAT  SUBSYSTEM.  MORE  SPECIFICALLY  WITHIN  INDIVIDUAL  LRU 


•SUBSYSTEM  MANUFACTURER  HAS  TOTAL  RESPONSIBILITY  AND 
CONTROL  OF  HIS  OESIGN 


•  HIGHER  ORDER  TESTS  OR  OTHER  SUBSYSTEMS  HAVE  LIMITED 
VISIBILITY  INTO  INDIVIDUAL  LRUs 


•  DEPENDENCE  ON  PROPER  PERFORMANCE  OF  OTHER  SUBSYSTEMS  LEADS 
TO  INCORRECT  FAULT  ISOLATION 
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Figure  4-23 


PILOT-VS-MAiNTENANCE  CREW  FAULT  DATA  READOUT 


*  TO  SIMPLIFY  THE  PILOT'S  TASKS  OF  INTERPRETING  FAULT  DATA, 
FAULTS  REPORTED  TO  THE  PILOT  SHOULD  BE  ALPHABETIC 
DESCRIPTIONS  INSTEAD  OF  CODED  MESSAGES. 


•  MAINTENANCE  CREWS  TYPICALLY  HAVE  TIME  TO  OECODE 
MESSAGES  AND  COOED  FAULT  REPORTS 


•  FAULT  DATA  CAN  BE  USEFUL  TO  THE  PILOT  TO  ASCERTAIN  WEAPON 
SYSTEM  CAPABILI'i  Y 


Figure  4-24 


MANAGEMENT  AND  CONTROL 

MANAGEMENT  EMPHASIS  IS  ESSENTIAL 

•  THE  CONTRACTING  AGENCY, THE  CONTRACTOR  AND  SUBCONTRACTOR  MUST 
RECOGNIZE  ST/BIT  AS  A  LEGITIMATE  FIRST  LINE  REQUIREMENT  WITH  THE 
SAME  LEVEL  OF  IMPORTANCE  GIVEN  PERFORMANCE  REQUIREMENTS 


EARLY  DESIGN  EMPHASIS 

•  TOTAL  SYSTEM  DESIGN  MUST  BE  COORDINATED  EARLY  WITH  INDIVIDUAL  SUB¬ 
CONTRACTORS  TO  ASSURE  COMPATIBILITY  OF  DESIGNS 

•DESIGN  ENGINEERS  MUST  BE  COMMITTED  TO  DESIGNING  TESTABLE  EQUIPMENT 
EARLY 

•ST/BIT  DESIGN  ANALYSES  MUST  BE  PERFORMED  FARLY;  PRIOR  TO  COMMITTING 
DESIGN  TO  HARDWARE 

•  DESIGN  SHOULD  CONTAIN  THE  MAXIMUM  AMOUNT  OF  FLEXIBILITY  TO  ACCOM¬ 
MODATE  DEVELOPMENT  CHANGES 
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Figure  *1-25 


DEMONSTRATIONS  AMD  TEST 


•  ST/BIT  SHOULD  BE  AN  INTEGRAL  PART  OF  SYSTEM,  SUBSYSTEM  AND 
LRU  TESTING 

•  INTEGRATION  TESTS  SHOULD  CONTAIN  SPECIFIC  ST /BIT  OBJECTIVES 

•  MAINTAINABILITY  DEMONSTRATIONS  SHOULD  BE  DESIGNED  AROUND 
ST/BIT 

•  LRU  TESTING,  QUALIFICATION  AND  BENCH  TESTS  SHOULD  CONFIRM 
ST/BIT  FUNCTION 

•  PERHAPS  A  SPECIAL  ST/BIT  TEST  SHOULD  BE  PERFORMED  INDUCING 
AT  LEAST  ONE  TYPE  OF  EACH  TYPICAL  FAULT  WITHIN  AN  LRU 

•  INTERMEDIATE  LEVEL  TESTS  SHOULD  CONFIRM  ST/BIT  RESULTS 

•  FIELD  FOLLOW  UP  A  MUST  -  ALL  POSSIBLE  CONDITIONS  AND  EVENTS 
IN  THE  FIELD  ARE  EXTREMELY  DIFFICULT  TO  PREDICT  AND  EACH 
INDIVIDUAL  EVENT  MUST  BE  TRACKED.  FIELD  EXPERIENCE  AND  SYSTEM 
MATURITY  GO  HAND-IN-HAND 

Figure  *4-26 

INTERMITTENTS 

•  INTERMITTENT  FAILURE  CONDITIONS  ARE  GENERALLY 
DIFFICULT  TO  ISOLATE 


•  ST/BIT  SYSTEMS  WITH  STORAGE  AND  HISTORY  PROVISIONS 
PROVIDE  VALUABLE  INSIGHT  INTO  LOCALIZING  AND 
ISOLATING  INTERMITTENT  FAILURE  OF  LRUs  IN  THE  SHOP 


«SUCH STORAGE  SHOULD  BE  A  SPECIFICATION  REQUIREMENT 
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Figure  4-27 


FIELD  SUPPORT  AND  EVALUATION 


THE  BEST  DESIGN  EFFORT  COUPLED  WITH  EXTENSIVE  ANALYSES  STILL  DO 
NOT  INSURE  100%  PERFORMANCE  WHEN  SYSTEMS  ARE  DEPLOYED.  THERE  IS 
A  MATURATION  PROCESS  THAT  ACCOMPANIES  THE  DEVELOPMENT  OF 
AVIONIC  SYSTEMS.  SYSTEM  COMPLEXITY  AND  NEW  TECHNOLOGIES  NECES¬ 
SITATE  A  MATURATION  PERIOD  FOR  BOTH  FSD  AND  PRODUCTION  PHASES 


•FIELD  TEAMS 

♦  Experienced  Systems  Engineers  On-Site  to  Evaluate  Events  and  Assess 
System  Performance 


•  OATA  COLLECTION  ANO  FOLLOW-UP 

•  Personnel  On-Site  to  Collect  Data  Are  Necessary 

•  Personnel  Assigned  to  Follow-Up  Problem  Areas  and  Effect  a  Solution 


Figure  4-28 

TRAINING  SIMULATORS 


•  WITH  THE  EXTENSIVE  USE  OF  MICROPROCESSORS  IN 
CHANGES  TO  SYSTEMS  (ECPs)  CAN  BE  HANDLED  TO  A 
BY  SOFTWARE  CHANGES 


HARDWARE, 
LARGE  EXTENT 


TRAINING  IN  THE  USE  OF  ST/BIT  DOES  NOT  REQUIRE 
EXTENSIVE  SIMULATIONS  OF  ALL  FLIGHT  PARAMETERS  AND  A tc 
ATTITUDES  REQUIRED  BY  SIMULATION  SYSTEMS 


•  USE  OF  THE  ACTUAL  AVIONIC  HARDWARE 
ANO  ECONOMICAL 


CAN  BE  MORE 


EFFECTIVE 
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Figure  *1-29 

BIT  CORRELATION  TO  OTHER  TEST  LEVELS 


•  TEST  RESULT  S  AT  THE  A/C  CAN  PROVIDE  VALUABLE  FAULT  INFORMATION 
FOR  FAULT  ISOLATION  IN  THE  SHOP 

•  TEST  TIMES  CAN  8E  REOUCEO  BY  ENTERING  TEST  PROGRAMS  AT  POINTS  THAT 
CORRESPOND  TO  FUNCTIONAL  FAILURES 

•INTERMITTENT  FAULTS,  HAVING  BEEN  DETECTED  BY  BIT,  CAN  BE  CONCENTRATED 
ON  IN  THE  SHOP 

•  EIT  CAN  BE  USEO  IN  THE  SHOP  TO  VERIFY  CORRECTIVE  ACTIONS 

•  TRFND  DATA  CAN  BE  ACCUMULATED  IN  THE  SHOP  TO  LOCATE  BAD  ACTORS  OR 
OVERALL  FLEET  PROBLEMS 


•  THE  AIR  FORCE  SHOULD  IMPLEMENT  PROCEDURES  TO  MAKE  FLIGHT  LINE  DATA 
AVAILABLE  TO  THE  SHOPS 


Figure  ^-30 

SUBSYSTEM  DESIGN  LESSONS 


•  SUBSYSTEM  VENDORS  SHOULD  PERFORM  SUBSYSTEM  FAULT  ANALYSIS  AND 
TESTING  TO  PROVIOE  INFORMATION  REGARDING  SYSTEM  DEGRADATION 
ASSOCIATED  WITH  EACH  FAULT  INDICATION 


•  A  SUBSYSTEM  SELF  TEST  DESIGN  SHOULD  KEEP  THE  SEQUENCE  OF  TESTS 
INDEPENDENT  OF  THE  SUBSYSTEM  MODE  TO  THE  EXTENT  POSSIBLE.  THIS 
FEATURE  IS  DESIRABLE  SINCE  IT  KEEPS  THE  LIMITATIONS  OF  MODE 
SELECTION  FROM  INHIBITING  THE  PERFORMANCE  MONITORING  OF  ANY 
TESTS 


•  WHILE  DEVELOPING  A  SUBSYSTEM'S  SELF  TEST  CAPABILITIES.  AN  ANALYSIS 
SHOULD  BE  MADE  EARLY  IN  HIE  DESIGN  TO  DETERMINE  THE  INTERDEPENDENCE 
OF  TESTS.  SHOULD  THE  DESIGN  REQUIRE  A  DEGREE  OF  INTERDEPENDENCE 
BETWEEN  TESTS.  A  HIERARCHY  SHOULD  BE  ESTABLISHED  TO  DETERMINE  THE 
ORDER  OF  THE  INDIVIDUAL  TESTS. 


•  WHERE  APPLICABLE,  THE  SAME  FAULT  INFORMATION  WHICH  IS  1  RANSMITTED 
TO  THE  CENTRAL  COMPUTER/STORAGE  DEVICE  SHOULD  BE  SIORED  WITHIN  A 
SUBSYSTEM'S  NON  VOLATILE  MEMORY 
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Figure  ^-31 

SUMMARY 


•  F-16  DESIGN  EMPHASIZED  ST/BIT 

*  FIELD  PERFORMANCE  IS  GOOD 

•  MANY  INNOVATIVE  FEATURES  WERE  INCORPORATED  INTO 
F  16  ST/BIT  DESIGN 

•  APPLICABLE  TO  OTHER  PROGRAMS 

•  F-16  LESSONS  LEARNED  PROVIDE  BASELINE  FOR  FUTURE 
SYSTEM  SPECIFICATION  AND  DESIGN 


F-16  AN/AP6- 66 
ST/BIT  SUCCESS  STORY 

Jim  Victor 
WESTINGHOUSE 


Mr.  Victor  is  the  manager  of 
maintainability,  BIT,  and 
safety  engineering  for 
Westinghouse,  DESC. 


HIGHLIGHTS  OF  THE  PRESENTATION 

The  capabilities  of  the  F-16  AN/APG  radar  system  are  seen 
in  both  the  Air-to-Air  and  Air-to~Surface  modes.  In  the  Air- 
to-Air  mode  these  include  downlook  detection;  uplook  detection; 
ACM/dogfight  autoacquisition;  manual  acquisition;  and  range, 
angle  and  velocity  track.  In  the  Air-to-Surface  mode  these 
include  real  beam  ground  map;  expanded  map;  Doppler  beam- 
sharpened  map;  air-to-ground  ranging;  sea  target  detection; 
beacon;  and  freeze.  AN/APG-66  operating  features  include: 
head-up,  hands-on  critical  switehology;  long-range  all-aspect 
detection  and  track;  clean  scope  display /downlook  target 
detection;  low  target  false  alarm  rate;  good  inherent  raid 
resolution;  multiple  autoacquisition  modes;  comprehensive 
multimode  ECCM;  high  resolution  ground  mapping/target  designa¬ 
tion;  sea  clutter  rejection  for  ship  detection;  accurate  AGR 
inputs  to  fire  control  computer  for  weapon  delivery;  high 
reliability/availability;  and  comprehensive  ST/BIT. 

Terminology  and  modes  of  operation  for  the  F-l6  radar 
ST/BIT  are  shown  in  Figure  5-1.  Ease  of  access  of  subunits  is 
emphasized  in  ST/BIT.  Self  test  is  defined  as  fault  detection; 
BIT  is  defined  as  fault  isolation. 

The  AN/APG-66  ST/BIT  maintenance  concepts  and  specifica¬ 
tion  requirements  are  shown  in  Figures  5-2,  5-3  and  5-A.  The 
mechanization  of  ST/BIT  on  the  radar  system  is  shown  in  Figure 
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5-5-  Figure  5-6  displays  a  simplified  block  diagram  and  5-7 
displays  the  F-16  A/C  readouts  of  ST/BIT  results. 


The  management  approach  to  ST/BIT  is  shown  in  Figure  5-8, 
and  5-9  displays  the  central  role  of  the  maintainability/ 
engineer  manager.  Overall  program  milestones  are  shown  in 
Figure  5-10,  and  Figures  5-11  through  5-16  display  the  ST/BIT 
maintainability  milestones.  Figure  5-11  is  a  condensed  over¬ 
view  of  the  major  milestones;  5-12  through  5-16  is  a  breakout 
of  these  major  points.  Figure  5-17  presents  the  in-house 
testing  and  demonstration  schedule  on  the  F-i6  radar  system, 
and  Figure  5-18  breaks  down  the  maintainability  growth  plan 
for  the  system.  Maintainability  trade-offs  are  shown  in 
Figures  5-19  through  5-22.  The  growth  of  self-test/built-in- 
test  experience  with  the  F-16  is  shown  in  Figure  5-23-  Block 
4  was  isolated  and  elaborated  on  in  Figure  5-24.  Software 
modifications  are  shown;  there  are  10  software  break-ins  in 
this  block,  thereby  reducing  the  number  of  hardware  changes 
required.  The  computer  has  34  K  words  of  memory,  of  which  4 
K  are  used  for  self  test.  A  total  of  108  key  parameters  were 
identified  as  being  required  for  ST/BIT  to  be  100  percent 
complete.  Completion  included  programming,  debugging  and 
verifying  the  parameter  test. 

Pre-demo  faults  were  inserted  for  the  internal  use  of 
Westinghouse  as  shown  in  Figure  5-25  to  give  confidence  to  a 
successful  demonstration.  The  number  of  faults  inserted  were 
related  to  the  expected  failure  rates  of  the  units.  The  total 
number  of  SRUs  was  79- 

The  results  of  the  0-level  maintainaeility  demonstration 
are  shown  in  Figure  5-26.  ST/BIT  production  improvements  for 
the  F-l6  radar  are  demonstrated  in  Figure  5-27.  Figure  5-28 
shows  the  data  analysis  and  corrective  action  approach  agreed 
to  by  an  evaluation  committee  which  provided  for  the  recogni¬ 
tion  of  problems  and  the  implementation  of  correction  action 


in  the  field.  The  committee  helped  to  fill  the  gap 
between  the  spec  requirements  (which  had  been  met)  and  the 
user  requirements.  The  program  for  improving  the  ST/BIT  was 
part  of  the  overall  AFTEC  test  program  and  w as  not  costed 
separately.  Block  B  (flying  in  the  MOT&E  aircraft)  problems 
are  identified  in  Figures  5-29  and  5-30.  The  corrective 
actions  taken  to  Block  B  problems  as  implemented  in  ECP-331 
are  shown  in  Figures  5-31  and  5-32.  Snowy  runways  and  squat 
switch  bounce  problems  were  pointed  out,  including  their 
effects  on  calibration  routines.  New  APG-66  ST/BIT  mechani¬ 
zations  in  ECP-331  are  indicated  in  Figure  5-33.  The  compari¬ 
son  in  supportability  between  the  Block  B  and  ECP-331  con¬ 
figurations  shows  that  in  some  cases  the  aircraft  flew  multi¬ 
ple  missions  with  the  same  fault  because  maintenance  was 
deferred  to  the  end  of  the  day. 

Results  and  conclusions  of  MOT&E  data  analysis  on  ECP-331 
configured  aircraft  are  shown  in  Figures  5-3*1  and  5-35  indicat¬ 
ing  that  some  faults  not  confirmed  by  BIT  may  indicate  trends 
in  system  degradation.  Figures  5-?6  and  5-37  delineate  F-l6 
lessons  learned;  it  is  noted  that  the  levels  of  fault  detec¬ 
tion  for  pilots  and  maintenance  should  be  treated  differently 
since  their  requirements  are  different.  Development  plan 
requirements  are  shown  in  Figure  5-38;  operational  development 
requirements  are  shown  in  Figure  5-39- 

Looking  to  the  future,  the  essential  steps  necessary  to 
successfully  fulfill  ST/BIT  requirements  are  shown  in  Figure 
5-40;  recommendations  for  an  effective  BIT  program  are  given 
in  Figures  5—^1  through  5—^9 - 

DISCUSSION  POINTS 

•  Inconsistencies  have  been  shown  between  pilot  reports 
and  0-level  maintenance  tests  as  well  as  between  suc¬ 
cessive  0-level  maintenance  tests.  BIT  software  is 
an  integral  part  of  the  operational  software. 
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•  Airlines'  experience  is  that  15  percent  of  the  shop 
failures  are  "hard"  failures  (e.g.  opens  and  shorts). 

It  was  questioned  whether  the  BIT  is  mechanized  to 
detect  "soft"  (e.g.  timing)  failures  and  intermit- 
tents.  Westinghouse  stated  that  it  has  set  parameter 
sensing  levels  to  sense  soft  failures. 

•  Trend  data  indicating  unit  deterioration  and  projected 
need  for  replacement  are  not  recognized  by  the  avionics 
maintenance  and  supply  system  at  the  present  time.  We 
should  investigate  the  use  of  this  trend  data. 

•  "Beyond  BIT"  maintenance  requires  a  high-skill-level 
technician. 

•  The  field  problem  is  not  the  detection  of  the  fault 
but  rather  too  many  false  alarms. 

•  The  problem  of  specifying  a  MTBR  is  that  removals  will 
be  held  to  a  minimum  rather  than  removed  as  required 
for  the  most  effective  maintenance. 

•  A  measurable  definition  of  "false  alarm"  is  required. 

In  the  EF-111,  for  example,  keying  of  the  HF  radio 
resulted  in  a  false  alarm  that  was  ultimately  worked- 
around-clearly — a  false  alarm. 
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Figure  5-1 


Sf  .f-Test/Built-ln  Test 
Terminoiogy  and  Modes  of  Operation 

•  Self-Test  Is  a  Continuous  Noninterruptive  Fault  Detection 
Function  That  Is  Mode  Oriented 

-  CPM:  Continuous  Performance  Monitor 

-  Nl:  Noninterruptive 

-  Offbar:  Tests  Conducted  at  Antenna  Turnaround 

•  Built-In  Test  Is  a  Hierarchical  Group  of  Interruptive  Tests 
That  Detect  and  Isolate  Failures  to  a  Single  LRU 

-  BITRadar:  Automatically  Initiated  at  System  Turn-On 

-  BIT  •  Pilot  Initiated  Via  A/C  Fire  Control  Nav 

Panel 


Figure  5-2 

F-16  AN/APG-66 

O-Level  Maintainability  Requirements 


Analysis 

Test 
or  Demo 

Inspection 

•  Fault  Detection  >95% 

•  Fault  Isolation  >95% 

V* 

•  Maintenance  Time 

s 

S 

.  Mean  (Met)  s  0.5  Hour 

.  MAX(Mmaxct)  <1.0Hour 

i  <  5%  False  Alarm 

V* 

»  No  Adjustments,  Alignments 
or  Calibrations  Permitted 

✓ 

•  No  Fhght  Line  AGC  Required 

* 

♦  Within  Skill  Level  Three 

Capability 

✓ 
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INTERMEDIATE  LEVEL  MAINTENANCE  CONCEPT  Figure  5-3 


•  LRU  Maintenance  Time 

-  Mean  (Mqt)  1.0  Hour 

-  Max  (Mjv]axct)  2.0  Hours 

•  Avionics  Intermediate  Test  Station  (AIS) 

-  9 7%  Isolation  to  Faulty  SRU 

-  Remove  and  Replace  Faulty  SRU 

-  Verify  Repair  of  LRU 

•  Within  Skill  Level  Five  Capability 

•  Faulty  SRU  Forwarded  to  Depot  for  Repair 

DEPOT  LEVEL  MAINTENANCE  CONCEPT  Figure  5-4 

•  SRU  Fault  Isolation  Requirements 

-  Single  Element  80% 

-  Two  Elements  85% 

-  Three  Elements  90% 

0  Depot  Test  Station 

-  Isolation  to  Component 

-  Remove  and  Replace  Faulty  Component 

-  Verify  Repair  of  SRU 

§  Repaired  SRU  Returned  to  Stores 


Figure  5-5 


Self-Test  Built-In-Tost 
Mechanization 


•  Self-Test  is  Used  to  Determine  the  Health  Status  of  the 
Radar 

-  Self-Test  Report  (6002)  to  the  FCC  Is  a  Single  Bit  Report 

-  Single  Self-Test  Fault  (6002)  Report  Can  Be  Set  as  a 
Function  of  as  Many  as  80  Different  Checks 

-  No  Means  Exists  for  Determining  Which  Self-Test  Check 
Failed 

•  Built-In-Test  Is  a  Fault  Isolation  Routine  that  Is  Interruptive 
and  Isolates  a  Failure  to  an  LRU 

-  Automatical  Initial  Turn-On 

-  Pilot  Initiated  Whenever  Derived 
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Figure  5-6 


F-16  A/C  Readout  of  ST/BiT  Results 
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Figure  5-8 

ST/BIT/M  Organizational  Interface 


Figure  5-9 

S.T./BIT/M  Engineer  Role 


•  Member  of  Design  Team 

•  Specializes  in  Maintainability  Requirements 

•  Relieves  Designer  of  Added  Maintainability  Design  Functions 

•  Maintainability  Engineer  Available  for: 

-  Impromptu  Discussions/Reviews 

-  Trade-Offs 

-  LRU  Partitioning 

-  Raw  Engineering  Data  Early 

(Parts  List,  Schematics,  T-Specs,  etc) 

-  Drawing  Sign-Off 
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Program  Milestones 


PROf  RAM  UUESTONES 
Sf  If  TCST-0IT  MAINTAINABILITY  TASKS 

!  saf  Yf  ST  RUUT  IN  TCST  REQUIREMENT  &  DESIGN 
>>SUM  FlOPAflAVETtN  .'SI 
MAINTENANCE  concept 
FUNCTIONAL  DIAGRAMS 
TEST  ftOWOlACRAMS 
S'  8>l  HAROWARE  REQUIREMENTS 
DRAWING  PEVlfWS/SlGNOEE 
TEST  TOLERANCES 

trade  studies 

DESIGN  REVIEWS 
SYSTEM  INTEGRATION 

2  SOFTWARE  PROGRAMMING  LIAISON 

TEST  MUNITION 
programming  now  Diagrams 

VAllljAllON  DEBUG 

ATI  AS  LANGU  4GE  INTER!  ACE  WITH  US 

3  SUE  TCSMUILT  IN  TEST  ANALYSIS  &  PREDICTIONS 

EFFECTIVENESS  PREDICTIONS  VIA  FM£A 
ILS  INTERFACE 

TOOLS.  SKILLS  LEVEL.  FACILITIES.  OEPTH  A 

frequency  of  repair 

4  MAINTAINABILITY  R  E  QUIRE  ME  NTS  &  QESGN 

MAINTENANCE  concept 

MAINTAINABILITY  CHECKLIST 

FUIE  PARA^tTER  LIST  'FUNCTIONAL  DIAGRAMS 

FLUE  PARTITIONING 

KST  «L*)W 0«*GAams 

TEST  POINT  »OENTlFiCAT«C^S 

karowar:  requirements 

ORaWING  SIGN-OFF 
TfST  TOLCRANCII 
Maintainability  obmb  reviews 

I  MAINTAINABILITY  ANALYSIS  B  PMOtCTlONl 
MTTR  PREOlCTtONBNtR  MU-HOBK-4T2 
STE/ETE'XlS  REQUIREMENTS 
FMEA  CORRELATION 
ORlA  inputs 

6  MAINTAINABILITY  DEMONSTRATION 
’1ST  Pi  ANNING 

ParamETERiCOUPONEnT  FAILURE SSEiECTEO 
ON  EQUIPMENT  DEMONSTRATION 

-  SltiARlLlTY/DUALlTV/f  AT  LIAISON 

I  US  ENGINEERING  4  CUSTOMER  LIAISON 

5  OAfA  ITEM  SUBMISSIONS 

^PROGRAM  PLAN 
M  PREO'CV,ONS 

QUANTI*  AllvE  MAINTAINABILITY  ANAI  fSlS 
ST  e  T  CONTROL  COCUMCM 
^DEMONSTRATION  PLAN 
ORlA 

M  ASSESSMENT  4  DEMONSTRATION  report 
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Figure  5-10 
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Figure  5-11 


Seif-Test/BIT/Maintainability  Milestones 


1977 


JFMAMJJASOND 


A  Sl*rt  /  c\ 


Figure  5-12 

Self-Test/BIT/Maintainability  Milestones 


1975 

1976 

1977 

N  O 

JFMAMJJASOND 

JFMAMJJASOND 

A.  System/LRU  Parameter  List 

B.  Maintenance  Concept 

C.  Functional  Diagrams 

D.  Test  Flow  Diagrams 

E.  ST/BIT  Hardware  Requirements 

F.  Drawing  Review/Sign  Off 

G.  Test  Tolerances 

H.  Trade  Studies 

I.  Design  Reviews 

J.  System  Integration 


Figure  5-13 

Seif-Test/BIT/Maintainability  Milestones 


2.  Software  Liaison 


1975 

1976 

1977 

N  D 

4FMAMJJASOND 

JFMAMJJASOND 

A.  Test  Definition 

B.  Programming  Flow  Diagrams 

C.  Validation  —  Debug 

D.  Atlas  Interface  With  ILS 


/s^SI.X  /c\  Co*nplftt»  /p\Pfetimioaiy  /fi* ' 


Figure  5  — 

Self-Test/u.  .'/Maintainability  Milestones 


Figure  5-15 

Self-Test/BIT/Maintainability  Milestones 


6.  M  Demonstration 


A.  Test  Planning 


B.  Parameter/Component 
Failures  Selection 


C.  Demonstration 


Figure  5-16 

Self-Test/BIT/Maintainability  Milestones 


9.  Data  Item  Submissions 


1975 

1976 

1977 

M  0 

JFMAMJJASOND 

JFMAMJJASOND 

A.  M  Program  Plan 

B.  M  Predictions 

C.  Quantitative  M  Analysis 

D.  ST/BIT  Control  Document 

E.  M  Demonstration  Plan 

F.  ORLA. 

G.  M  Assessment  and 
Demonstration  Report 


R\  Update  every  00  Days  as  Required 


Update  Every  00  Days  as  Required 


/s\31»rt  /c\Comp'»lt  /p\PiBllmlmiy  /r\  n#vlje 
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Figure  5-17 


Figure  5-19 


Maintainability  Trade-Offs 


•  STALO  Multiplier  SRU 

•  Pressurization  System 

•  LVPS  Distribution 

•  RCP  BIT  Board  Elimination 


Figure  5-20 


Waveguide  Pressurization 
Trade  Study 


Absolute  Pressure  System 


Cost 


$620  Life  Cycle  Cost 
Savings  $160 


Reliability  Less  Average  Seal  Stress 
(OtolOpsia*  vs 
7to10psia*  *  ) 

S'ze  117  cu  in.  Smaller 

(84  vs  201  cu  in.) 

WmgW  0.5  lbs  Lighter 

(3.3  lbs  vs  3.8  lbs) 

*  Only  at  High  Altitudes 
•  •  All  Altitudes 


Gauge  Pressure  System 

Permits  On-Ground  Self-Test/BIT When 

•  A/C  Engine  Air  (Servo  Air) 

or 

•  AGE  (Pressurization  Bottle) 

Is  Provided. 
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Figure  5-21 

Waveguide  Pressurization 


It  0104  GBS 


Figure  5-22 

Distributed  LVPS  Trade  Study  Summary 


•  Reduces  Built-in-Test  Complexity 

•  Improves  Maintainability  Through  Easier  Troubleshooting 

•  Reduces  Factory  and  AGE  Test  Equipment  Complexity  and 
Cost 

•  Reduces  LRU  Interconnection  Cabling  and  Cable  Wiring 
Protection 

•  One  Less  LRU  To  Be  Maintained,  Handled,  and  Spared 

•  Reduces  Interface  and  Harmonization  Problems  at  System 
Level 

•  Reduces  Weight,  Component  Count  (By  7%),  and  Cost 
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Figure  5-2*1 


FSD  Software 
Modifications  & 
Block  Update  Summary 


Configuration 

BIP 

System  Performance 
Patches 

ST/ BIT 
Patches 

Block  4A 

157  Patches 

42  Patches 

Block  4B 

31  Patches 

9  Patches 

Block  4C 

25  Patches 

7  Patches 

Block  4D 

17  Patches 

9  Patches 

Block  4E 

85  Patches 

34  Patches 

Biock  4F 

16  Patches 

10  Patches 

Block  4G 

0  Patches 

3  Patches 

Block  4H 

32  Patches 

6  Patches 

Block  41 

35  Patches 

11  Patches 

Block  4J 

7  Patches 

3  Patches 

Subtotal  405  Subtotal  1 34  j 


Total  539 


Figure  5-25 

Pre-Demo  Fault  Insertion  Status 


No.  of  SRUs 
Faulted 

Faults  Detected 
Faults  Inserted 

No.  of  Tests 
Exercised 

Antenna 

5 

54/58 

13 

LPRF 

11 

80/120 

30 

Transmitter 

8 

27/33 

14 

DSP 

27 

958/1090 

19 

Computer 

13 

06/103 

17' 

RCP 

3 

43/60 

6 

Total 

67 

1 248  _  pro/ 
1462- 85/“ 

99 
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Figure  5-26 


F-16  FCR  0-Level  Maintainability 
Demonstration  Requirements  and  Results 


•  Maintainability  Demonstration  Requirements 

-  150  Faults  To  Be  Inserted  Throughout  Six  LRUs  and 
Distributed  According  to  Relative  Failure  Rates 
(Selected  Via  Random  Number  Generator) 

-  Each  SRU  (79  Total)  Shall  Have  at  Least  One  Failure 
Inserted 

*  Maintainability  Demonstration  Resuits 

Self-Test  Fault  Detection 
94% 

Built-In-Test  Fault  isolation 
98% 


Figure  5-27 

AN/APG-66  Production 
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Figure  5-28 


Joint  (W)/G.D./USAF 
Agreement  as  a  Result 
of  Meeting  at  HAFB 

(7-26-79) 

Agreed  Upon  Approach  of  Increasing  Catastrophic  Faults 
and  Labeling  Them  as  1010  Reports  on  FCNP. 


Agreed  Upon  Eliminating  6002  Reports  from  MCL  and 
Retaining  6002, 1010,  BIT  Reports  and  Self-Test  Fault  Flag 
Filter  in  the  MFL  Recall  Function. 


Agreed  Upon  a  Joint  (MOT&E,  SPO,  AFTEC,  G.D.,  (W) 
Committee  Evalutlon  of  MOT&E  Results  Periodically. 

-  Phase!  Phase!?  Phase  III 


Figure  5-2Q 


What  Were  the  Pilot  Noted  Problems 
Affiliated  With  Self-Test  and 
Built-in  Test?  (Block  B) 


*  Self-Test  Reports  Occurred  In-Flight  Too  Often, 
Illuminated  the  Master  Caution  Light,  and  the  Pilot 
Debrief  Report  Stated: 


-  Frequently  the  Radar  Worked  OK 


Frequently,  Can't  Run  BIT  at  Time  of  Self-Test 
Failure 


-  Frequently  No  Failures  Noted  When  BIT  Was  Run 


-  Conclusion:  Self-Test  Was  of  No  Value  to  the  Pilot 


Figure  5-^0 

APG-66  ST/BIT  Problems  Incurred  at  HAFB 
With  Block  B  Configuration 

•  Self-Test  Reports  Do  Not  Provide  an  Indication  of  Which 
LRU  is  Defective 

•  Frequently  Postfllght  Maintenance  Fails  to  Reproduce 
In-Flight  ST  Reports 

•  Self-Test  Problem  Created  the  Following  Situations: 

-  Pilot  Has  Little  Confidence  In  Self-Test  or  Built-In  Test 

-  Many  Maintenance  Hours  Spent  Trying  to  Duplicate 
Seif-Test/Bullt-ln  Test  Reports 

-  Many  CND’s  and  RTOK’s  Due  to  “Shot-Gun”  Removal  of 
LRU’s 

•  Built-In  Test  Is  Not  a  Problem 

-  LRU’s  Removed  for  BIT  Report  Are  Usually  Defective 
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Figure  5-31 


APU-66  Improvements  In  ECP-331  That 
Correct  Block  B  ST/BiT  Problems 


Computer  Hardware  Changes  to  Eliminrte  Hang-Up 
Condition  (1005) 

Antenna  Flipper  Hardware  Change  and  Retrofit 

System  OFP  Software  Changes  That  Affect  ST/BIT 

-  System  Calibration  Routines  Modified  To  Meet  Operational 
Scenario 


-  Transmitter  Protection  and  Control  Logic  Modified 

-  Antenna  Overtemperature  Logic  Modified 

-  A/C  Interface  Modifications  -  Squat  Switch  Bounce 

-  ECS  Shutdown 

-  INU  Down  Accounted  for 

-  DSP  Initialization  Header  Word  Fix 

*  Computer  Hang-Up  Restart  Logic  Modified 


Figure  5-32 


APG-66  Improvements 
in  ECP-331  That  Correct 
Block  B  ST/BIT  Problems 

•  ST/BIT  Software  Corrections  and  Improvements  Summary: 

•  Transmitter  Peak  Doted  Failure 

-  Transmitter  High  Voltage  Failure 

-  Transmitter  Calibration  Failure 

-  MSI  2001  Report 

•  Antenna  Drive  Defective 

-  Synchronizer  Test  Failure 

-  TX  Peak  Detector  Failure  at  TOF/from  STBV 

-  ST  Reports  Before  Required  Warm-up 

•  BIT  6450  Reports  at  Initial  Turn-on 

-  Transmitter  Peak  Detect  Failure  In  DBS  Mode 

-  MSL2001  Report  at  Radar  Turn-on 

•  MSI  2001  Reports  During  Radar  Self-Test 
■  Intermittent  Receiver  Verification  Failure 

-  Sell-Test  Failures  Not  Duplicable  on  Ground 

•  Only  Catastrophic  Faul  Reports  Illuminate  MCL 


Figure  5-33 


Changes  Included  in  ECP-331  Regarding 
Seif-Test  Reporting  System 

WEC  Increased  Catastrophic  Fault  Reports  from  4  to  31  Tests 


WEC  Provides  6  Words  (80  Bits)  to  the  FCC  In  Self-Test, 
Indicating  the  Failed  Parameter 


GD  Eliminated  ST  Report  (6002)  from  illuminating  the  A/C 
Master  Caution  Light 


GD  Illuminates  the  Master  Caution  Light  as  a  Function  of 
Catastrophic  Faults,  Only  (1010) 


MFL  Continues  to  Store  1010’s,  6002’s,  BIT  Report  Numbers, 
and  the  New  ST  Fault  Flag  Reports 

-  New  ST  Fault  Flag  Reports  Stored  in  Separate  List  (RST) 


Figure  5-3*1 

Results  and  Conclusions  of  MOT  &  E 
Data  Analysis 
(3-80  to  12-80) 


1317  Total  Flights 


Conclusion: 

&  ' 

•  Previous  Nulslr.ee  Indicators  of  ST  Failures 
Clearly  Resolved 


Figure  5-35 


Results  and  Conclusion  of  MOT  &  E  Data 
(9  A/C  from  3/80  thru  12-80) 


Conclusions:  •  Worst  Case  -Only  11.5%  of  Flights  Required  STFF  Usage 

•  Once  MFL  Report  Is  Cleared  by  LRU  Removal,  Everything  Is  Super 

-  Performance  -  Reliability 

-  Availability  -  ST /BIT 

•  STFF  13  an  Early  Indicator  of  System  Degradation 

Figure  5-3b 

r-16  AN/APG-66 
Specification  Lessons  Learned 

•  Maintainability  Specification  Requirements  Should 
Consider  Operational  User’s  Inputs  and  Needs 

-  Pilot  Needs 

-  Maintenance  Technician  Needs 

•  Method  of  Validating  Requirements  Should  Include  Field 
Operational  Usagfe 

*  Requires  New  Specs 

-  Requires  Field  Dedicated  Personnel  and  Asset  s  and 
User  Organization  Similar  to  AFTEC 

•  Consideration  Should  Be  Given  to  Intermittent  Malfunctions 
(Not  Duplicatable  on  Ground)  and  Effect  on  Higher 

Level  Maintenance 
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r'igure  5-37 


F-16  AN/APG-66  Specification 
Lessons  Learned 

Seif-Test/Built-in-Test  False  Alarm  Rate  Requires  Following: 

-  Clear  Definition  of  Same 

-  Practical  Requirement,  Taking  Into  Consideration  Other 
Parameters  Like  fault  Detection/Fault  Isolation 
Effectiveness  and  System  Reliability. 

-  Practical  Method  of  Verification 

Figure  5 -38 

F-16  AN/APG-66  Development 
Plan  and  Lessons  Learned 

Comprehensive  Detail  Planning,  Scheduling,  and  Monitoring 
Should  Include: 

-  ST/BIT  Development,  Integration,  and  Check-Out 
Concurrent  With  System  Performance  Development 

-  Assignment  of  Personnel  and  Equipment  Asset  s 

-  Pre-Demo  and  In-House  Effectiveness  Monitoring 

:  Institute  Test/Fix/Retest  Program 

-  Mil-Std-470  Should  Be  Revised 

Above  Elements  Are  Necessary  Toward  Obtaining  Opera¬ 
tional  Requirement  But  the  Task  Is  Not  Yet  Complete 
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Figure  5-39 


F-16  AN/APG-66 
Operational  Development 

•  Successful  Demonstration  of  Requirements  Is  Not  Attained 

Unfit: 

-  it  Works  as  Intended  With: 

A.  User  Personnel 

B.  User  Equipment  Asset  .s 

C.  User  Tech  Orders 

D.  Actual  Environmental  Conditions 

E.  Actual  Operational  Scenario 

Figure  5-^0 

Essential  Steps  Necessary  to 
Successfully  Fulfill  ST/ BIT  Requirements 


•  Program  and  Engineering  ManagementCommittment 

•  ST/ BIT/ Maintainability  Engineers  Must  Be  an  Integral  Part  of 
Design  Team 

•  Detailed  ST  I  BIT  Program  Plan  Development  Required  To  Be 
Integrated  With  System  Development  Program  Plan 

•  System  Assets  Need  To  Be  Committed  to  ST /  BIT 
Development 

•  Engineering  Personnel  and  Equipment  Assets  Required  for 
Extended  Time  Duration  After  Initial  Field  Deployment 

•  Joint  Plan  Involving  Contractor,  SPO,  MOTE,  and  Tactical 
Units  Required  for  Data  Collection  and  Field  Evaluation 
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Figure  5-^1 


Main  Program  Elements 


•  Definitive  Specifications 

•  Effective  Contractor  Program 

•  Formal  Measure  of  Achievement 

•  Field  Feedback  and  Problem  Correction 


Figure  5-^2 

Specifications 


Fault  Detection 

-  Probability  of  FD  for: 

:  Operational  -  Expanded  Tolerance  Limits 
e.g.,  ±  3  dB 

:  Malntenance/Pref  light  -  Minimum  Acceptable  Limits 
e.g.,  ±  1  dB 

-  Define  Means  of  Annunication  of  a  Fault 

-  Define  Class  of  Faults  To  Be  Reported 

e.g.,  Only  Hard  Faults, 

M  of  N  Transient  Faults, 

All  Faults. 

-  Define  Requirement  for  Data  Capture  of: 

:  Faults  Not  Reported  Ouring  Operation 
:  Operating  Conditions  at  Time  of  Fault 

e.g.,  Altitude,  Temp,  Mode,  G's,  Time,  elc 
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Figure  5-^3 


Specifications  (Continued) 


Fault  Isolation 

-  Probability  of  Unambiguous  FI  to:  1  LRU 

JLLRUs 


-  Define  LRU 

:  Large  Box  -  3  Level  Maintenance 
:  Plug  in  Assembly  -  2  Level  Maintenance 
:  Discard  Element  - 1  Level  Maintenance 

-  Define  Beyond  BIT  Maintenance  for  Each  Assembly 


Figure  5-hl\ 

Specifications  (Continued) 


False  Alarms 

-  Define  a  False  Alarm  * 

(An  Intermittent  In  an  Essential  Circuit  Should  Not  Be  Classified  as  a 
False  Alarm) 

-  Express  Requirement  as  False  Alarm  Rate 

e.g.,  F.A.  Shall  Not  Exceed  0.01/Mlsslon  Gper.  Hr. 

-  Define  Requirement  for  Transient  DataCapture 
Reconfiguration 

-  Specification  Minimum  Allowable  Probability  of  Mission  Success  Which 
Includes  Reconfiguration  Effectlvlty 

■  Specify  Reliability  Maturation  Status 

l.e.,  What  Fraction  of  Predicted  Reliability  Value  Is  the  Basis  for 
Field  Use? 
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Figure  5— 5 


Specifications  (Continued) 

Trade  Studies 

-  Define  Criteria -LCC 

-LSC 

-  Acquisition  Cost 

Qualification  Test  Requirements  *  See  Separate  Chart 
Data  items 

-  BIT  Program  PJan  -  Submit  90  DAC 

-  Allocation,  Analysis  and  Assessment  Report  -  Submit  90 
DAC  Update  Quarterly 

-  BIT  Growth  Plan  -  Submit  at  PDR 

-  BIT  Qualification  Plan  -  Submit  90  DAC 

-  Final  Design  Report 

-  Field  Feedback  and  Corrective  Action  Plan  r-Submit  at 
CDR 

-  Field  Effectiveness  Report  -  Quarterly 

Figure  5-^6 

Contractor  Program  Definition 

BIT  Program  Plan 

Define  -  Contractors  BIT  Design  Organization 

-  Analysis/Prediction  Techniques 

-  Design  Monitoring  Process 

-  Allocation  of  Requirements  to  Subordinate 
System  Elements 

-  Allocation  of:  Resources 

:  Manpower 
:  Prime  Equipment  Time 
:  Money 

-  Milestones 

-  Configuration  Control  -  Hardware 

-  Software 
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Figure  5-^8 


Contractor  Program  Definition 


BIT  Growth  Plan 

Define  -  Maturation  Process 

(Recommend  Randomly  Selected  Fault 
Insertion) 

-  Resources  in  a  Test/Fix/7 est  Iteration  Process 

-  Milestones 

-  Objectives  To  Be  Met 

-  Corrective  Process 


Figure  5— 8 


Formal  Measure  of  Achievement 

BIT  Qualification  Test 

-  Specify  Test  Sample  Size  -  Must  Be  Statistically 
Significant 

-  Define  Sample  Selection  Process  -  Failure  Weighted 

-  Random  Selection 

-  Define  Evaluation  Team  Personnel 

-  Specify  Pass/Fail  Criteria  for  Each  Parameter 

-  Specify  Evaluation  Test  Process  for  Such  Parameters 

:  False  Alarm  Rate 

:  Circuit  Elements  In  Which  a  Simulated  Fault  Could  Be  Catastrophic 
e.g.,  A  Short  on  a  90  KV  P.S.  Could  Destroy  the  Entire  Unit 

-  Define  Liability  of  Contractor  If  Test  Does  Not  Achieve 
Minimum  Limits 
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Figure  5-49 


Field  Feedback  and  Problem  Correction 

•  Define  User/Producer  Field  Evaluation  Team 

•  Define  Evaluation  Team  Objectives 

•  Define  Team  Member  Responsibilities 

•  Define  Team  Evaluation  Criterion 

•  Define  Problem  Correction  Process 

•  Define  Problem  Correction  Liability  to  the 
Contractor 
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F/A-18A  AND  TF/A-18A 
AVIONICS  BIT 

Bob  Drummond 
MCDONNELL  DOUGLAS 

Mr.  Drummond  is  a  Section 
Chief,  Electronics,  at 
McDonnell  Douglas 

HIGHLIGHTS  OF  THE  PRESENTATION 

The  presentation  covered  the  following  six  aspects  of  F/A- 
18a  and  TF/A-lSA  avionics  BIT:  objectives;  integration  and 
design  considerations;  BIT  specification  requirements;  system 
description;  design-development-verification  process;  and 
status.  It  is  noted  that  the  internal  mechanization,  includ¬ 
ing  BIT,  of  the  fighter  (F)/attack  (A)  aircraft  is  the  same  and 
requires  no  internal  hardware  or  software  changes  when  recon¬ 
figuring  between  "fighter"  and  "attack." 

The  major  objectives  of  avionics  BIT  are  shown  in  Figure 
6-1.  In  meeting  these  objectives,  the  following  fou*  results 
were  obtained.  Though  not  a  specified  requirement,  BIT  may  be 
run  by  one  person,  resulting  in  additional  manpower  savings. 
Organization  Level  GSE  has  been  eliminated  and  I-level  fault 
isolation  (FI)  has  been  simplified  by  initiating  test  in  the 
failed  area  identified  by  BIT,  rather  than  initiating  a  com¬ 
plete  subsystem  test.  Cooling  fans  automatically  initiated 
during  ground  maintenance  permit  ground  operation  without 
external  support  equipment  up  to  103°F  (above  that,  the  system 
automatically  shuts  down).  BIT  code  readout  of  the  maintenance 
monitor  panel  can  be  activated  using  aircraft  battery  power. 
In-flight  presentation  of  the  weapon  system  status  combined 
with  automatic  mode  reversions  demonstrate  the  potential  for 
a  significant  increased  mission  effectiveness. 

A  summary  of  BIT  integration  and  design  considerations  is 
shown  in  Figure  6-2.  Off-the-shelf  equipment  comprises 
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approximately  35  percent  of  avionics  equipment  and  requires 
unique  integration  interfaces.  The  BIT  failure  criteria  imple¬ 
mented  to  minimize  false  alarms  is  summarized  in  Figure  6-3  and 
includes  thresholds  and  time  delays.  The  operational  flight 
envelope  and  environment  must  be  considered  and  experienced  in 
developing  these  thresholds  and  delays.  In  the  aircraft,  for 
example,  several  MUX  bus  faults  in  succession  on  a  channel  must 
take  place  prior  to  declaring  a  failure. 

The  development  of  BIT  thresholds  by  vendors  and  MCAIR 
generally  began  with  developing  tight  tolerances  and  then 
expanding  them  as  guided  by  in-flight  BIT  performance  results 
achieved  while  operating  in  the  operational  and  environmental 
conditions,  as  illustrated  in  Figure  6-4.  There  are  4l  WRAs 
which  contain  BIT  in  the  fighter  aircraft  configuration  and  58 
in  the  attack  cc if iguration  (with  2  pods).  Failures  are  re¬ 
ported  in  the  cockpit,  on  the  WRAs  themselves,  and  one  on  the 
MMP  ir,  the  nose  wheel  well.  Tolerances  for  periodic  BIT  and 
initiated  BIT  are  the  same;  however,  the  initiated  BIT  employs 
more  extensive  tests.  Automatic  reversion  to  a  degraded  mode 
or  manual  mode  is  performed  upon  declaration  of  a  failure. 
Selected  override  (emergency)  provisions  are  included  and 
maintenance  indications  of  the  override  are  retained  in  memory. 
Widening  BIT  tolerance  values  unjustifyably  may  make  BIT 
initially  look  good  but  will  not  result  in  an  effective  BIT 
system. 

BIT  time  delay  considerations  are  sir  in  Figure  6-5, 
seeking  the  bottom  of  the  bell  curve  as  an  ptimum  condition. 

A  summary  of  the  BIT  requirements  imposed  on  the  aircraft, 
as  well  as  the  requirements  imposed  on  the  suppliers,  are 
shown  in  Figure  6-6.  The  requirement  to  detect  98  percent  of 
the  equipment  failures  applies  to  individual  major  avionics 
subsystems.  Depending  upon  the  subsystem,  periodic  BIT  is 
performed  every  200  milliseconds  to  30  seconds.  An  initial 
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analysis  was  required  from  each  vendor  to  demonstrate  their 
designs  capability  to  detect  98  percent  of  the  failures. 

General  P/A-18A  status  monitoring  interfaces  are  illus¬ 
trated  in  Figure  6-7-  A  total  of  19  air-to-ground  and  13  air- 
to-air  tactical  parameters  are  monitored  for  training  purposes, 
as  well  as  certain  airframe  and  engine  parameters  to  measure 
stress  and  performance  trends.  Avionics  (NONMUX)  equipment 
BIT  interface  is  via  the  CSC  to  the  MC  and  anionic  MUX 
compatible  eqipments  interface  directly  with  the  MC .  Display 
to  the  pilot  is  in  real  time;  i.e.,  for  an  intermittent ,  the 
BIT  indication  will  come  and  go.  The  maintenance  monitor  panel 
in  the  nose  wheel  well  provides  a  stored  3  digit  BIT  code 
retrievable  by  ground  maintenance  personnel. 

BIT  operating  characteristics  are  shown  in  Figure  6-8. 
Initiated  BIT  may  be  activated  by  the  pilot  or  a  maintenance 
person.  Initiated  BIT  of  systems  such  as  armament  and  flight 
controls  is  not  permitted  in  flight  for  safety  reasons. 

Overall  status  monitoring  capability  for  each  avionic 
system  is  illustrated  in  Figure  6-9-  Cautions  and  advisories 
(e.g.,  being  interrogated  on  Mode  IFF  and  not  replying)  as 
well  as  BIT  information  are  displayed  to  the  pilot. 

The  Multipurpose  Display  provides  BIT  status  information 
to  the  pilot  from  most  subsystems  as  indicated  in  Figure  6-10;- 
it  also  serves  as  a  Control  Panel  for  initiated  and  maintenance 
BIT  and  memory  inspect . 

The  Maintenance  Monitor  Panel,  located  in  the  nose  wheel 
well,  can  handle  up  to  990  different  fault  codes  and  can 
store  up  to  62  at  any  one  time.  Less  than  300  fault  codes  are 
currently  employed.  Pressing  the  display  button  commands 
display  of  each  "tripped"  fault  code  for  1.5  seconds.  Releas¬ 
ing  the  button  leaves  the  code  on  for  10  seconds,  prior  to 
shut  off.  These  fault  codes  represent  the  4l  boxes  in  the 


lighter  Aircraft  Configuration  and  58  in  the  Attack  Configura¬ 
tion  plus  other  related  functional  failures.  Figure  6-11 
illustrates  selection  of  maintenance  BIT  for  the  SMS . 

Initiated  BIT  tests  vary  from  50  milliseconds  to  150 
seconds  in  the  longest  case  except  for  special  tests  such  as 
INS  gyro  bias  setting.  Most  times  are  between  2  and  15 
seconds . 

Memory  inspect  capabilities  which  are  in  the  full  scale 
development  aircraft  and  are  being  considered  for  production 
are  shown  in  Figure  6-12.  The  MC  is  used  to  examine  the 
memory  of  units  that  are  peripheral  to  the  MC  and  display  data 
for  the  address  selected.  This  memory  inspect  capability 
requires  the  use  of'T.O.s  and  maintenance  personnel  who  are 
trained  ir.  this  type  of  maintenance.  If  the  proper  aircraft 
parameters  are  recorded  during  the  occurrence  of  failures, 
potential  benefits  to  identify  intermittents  and  environmentally 
induced  failures  can  be  achieved.  These  failures  otherwise 
may  have  resulted  in  CNDs  and  RETOKs.  This  would  permit  SRAs 
with  intermittents  to  be  removed  from  the  flight  environment 
until  repaired,  avoiding  future  occurrence  these  problems  in 
"flight-status"  aircraft. 

Figure  6-13  shows  the  BIT  assurance  elements  in  the  F/A- 
18A  design,  development,  verification  and  production,  phases. 
Production  sell-off  tests  will  use  the  BIT  capabilities  of 
the  system  for  maintenance  support. 

Equipment  initial  BIT  assessment  test  requirements  and 
results  are  shown  in  Figure  6-1*1.  Requirements  are  shown  in 
the  lower  left  hand  corner  of  the  figure.  The  tests  are 
designed  to  provide  early  confidence  in  the  BIT  design.  Suc¬ 
cessfully  passing  the  test  would  provide  about  a  90  percent 
confidence  level  of  the  design  achieving  the  specified 
requirements.  Percentages  are  for  faults  both  detected  and 
isolated . 
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The  vendor  maintainability /BIT  demonstration  status  on 
production-configured  equipment  is  shown  in  Figure  6-15. 

Figure  6-16  indicates  the  status  of  BIT  development  from  flight 
test  program  data.  Generally,  the  criterion  for  "no  known 
BIT  design  problems"  is  30  flights  vrithout  reoccurrence  of  the 
problem. 

Figure  6-17  presents  the  FSD  F/A-18A  growth  trend  of  in- 
scheduled  0-level  maintenance  in  terms  of  MMH  per  FH,  currently 
showing  between  four  and  five  MMH/FH.  Maintenance  is  performed 
oy  contractor  personnel;  although  this  will  probably  degrade  in 
fleet  use,  the  many  maintainability  features  of  the  F/A-18A 
show  promise  of  high  payoff. 

Figure  6-l8  displays  the  FSD  F/A-18A  mean  flight  hours 
between  failures  showing  a  progression  from  1.8  to  8.4  flight 
hours.  Flights  are  realistic  flights  and  include  typical 
operational  type  missions.  On  the  average  it  takes  4.3  un¬ 
scheduled  0-level  maintenance  manhours  per  flight  hour  to 
maintain  the  aircraft  in  active  flight  status. 


DISCUSSION  POINTS 

•  Failure  recordings  in  the  F/A-18A  development  aircraft 
include  a  snapshot  of  the  failed  function  and  aircraft 
conditions  at  the  time  of  failure.  This  proved  very 
useful  in  BIT  development  and  could  be  useful  in 
production  for  troubleshooting  intermittent  and 
environmentally  induced  failures. 

•  A  significant  lesson  learned  from  F-15  BIT  was  to  not 
immediately  latch  BIT  indicators  but  rather  to  employ 
appropriate  performance  thresholds,  time  delays,  and 
failure  criteria  logic  prior  to  declaring  "failure". 

•  The  ability  to  capture  intermittents  and  environmentally 
induced  failures  would  significantly  increase  the  use¬ 
fulness  of  BIT. 

•  Intermediate-level  test  compatibility  with  BIT  requires 
compatibility  of  thresholds  between  BIT  and  I-level 
testing  and  the  elimination  of  test  voids  where  possi¬ 
ble  (including  verification  of  rhe  BIT  mechanization). 
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•  At  I-level,  BIT  should  be  the  final  test  before 
returning  the  unit  for  aircraft  service. 

•  Technology  has  permitted  most  avionics  subsystems  to 
be  mechanized  in  fewer  boxes.  Radars,  formerly  13- 
box  systems,  have  been  reduced  to  1-  to  7-box  systems 
With  these  boxes  becoming  more  complex,  they  are  more 
difficult  to  troubleshoot  even  though  there  are  fewer 
of  then.. 

•  A  concept  of  Peacetime/Wartime  BIT  failure  criteria 
levels  should  be  studied  for  potential  future  appli¬ 
cation. 

•  Reductions  in  false  BIT  failure  declarations  cm  be 
achieved  by  obtaining  better  quality  development  data 
and  then  properly  widening  tolerances  in  thresholds 
and  time  delays. 

•  Some  redundancy  (but  additional  unnecessary  expense) 
is  incurred  by  requiring  failure  indicators  on  each 
WRA  in  aircraft  that  have  a  central  system  failure 
indication. 

•  Though  the  scope  that  future  avionics  plays  in  non- 
avionic  areas  such  as  power  control  and  electro- 
hydraulic  is  not  specifically  defined,  it  is  defi¬ 
nitely  on  the  increase  and  BIT  requirements/design 
need  integration  in  parallel  with  the  system  require¬ 
ments/design  . 

•  The  question  of  determing  when  a  subsystem's  BIT  is 

ready  to  go  into  development  in  the  aircraft  should 
include  a  consideration  of  the  percentage  design 
complete  and  percentage  of  the  software  program 
verified .  / 


F/A-18A  AND  TF/A-18A  AVIONICS 

BUILT-IN-TEST  OBJECTIVES  Figure  6-1 

•  INCREASE  AIRCRAFT  AVAILABILITY 

-  DECREASE  DOWNTIME/SHORTEN  TURNAROUND  TIME 

-  AUGMENT  SYSTEM  MISSION  RELIABILITY  (HIGHER  FLIGHT  HOURS/EQUIPMENT 
OPERATING  HOURS) 


MAINTENANCE  APPROACH 
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•  MODERATE  SUPPORT  (LIFE  CYCLE)  RESOURCES 

-  MANPOWER 

-  TRAINING  REQUIREMENTS 

-  PERSONNEL  SKILLS  LEVELS 

-  TECHNICAL  MANUALS 

-  GROUND  SUPPORT  EQUIPMENT 

•  SIMPL IFY  AVIONIC  SUPPORT  ON  GROUND  AND  CARRIER 
FLIGHT/HANGAR  DECKS 

-  ELIMINATE  ORGANIZATIONAL  GSE 

-  REDUCE  DECK  ACTIVITY 

•  ENHANCE  AIRBORNE  MISSION  EFFECTIVENESS 

-  WEAPON  SYSTEM  CAPABILITY  STATUS 

-  AUTOMATIC  MODE  REVERSIONS 

-  AIR  VEHICLE  SAFETY  ADVISORIES 


AVIONICS  BUILT-IN-TEST 
INTEGRATION  ASPECTS  AND 
DESIGN  CONSIDERATIONS  Figure  6-2 


•  BIT  SYSTEM  ARCHITECTURE  c  SYSTEM  SAFETY 


HUMAN  ENGINEERING 


*  BIT  THRESHOLDS 
AND  TIME  DELAYS 


•  BEYOND  TRADITIONAL 
BLACK  BOX  BIT 


•  BIT/INTERMEDIATE  LEVEL 
TEST  COMPATIBILITY 


•  SELECTED  OPERATOR  •  SYSTEM/SUBSYSTEM  IMPACTS 

DETECTION/ISOLATION  vs  BIT 
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BIT  TIME  DELAYS 


Figure  6-5 


MMHs 


MISSION  CAPABILITY  IMPACT 
THRESHOLDS 


INEFFECTIVE  BIT  . 

EXCESSIVE  FALSE 

ALARMS 

PILOT  UNAWARE 

LOW  8IT 

CREDIBILITY 


INEFFECTIVE  BIT 
PILOT  IS  FAULT  DETECTOR 
MAINTENANCE  MAN  IS 
FAULT  ISOLATOR 
LOW  BIT  CREDIBILITY 


"TOO  SHORT  ^ 


-TOO  LONG 


TIME  SINCE  FAULT  DETECTED 


F/A-18A  AND  TF/A-18A  AVIONICS 
BIT  SPECIFICATION  REQUIREMENTS  Figure  6-6 


AIRCRAFT  SYSIEM  SPECIFICATIONS 
(S D-565,  AR-10) 


•  DETECT/ISOLATE  FAILURES 
(PILOT,  WRA.  M  PANEL) 

•  ACTIVATE  AT  EQUIPMENT  TURN  ON, 
MANUAL  COCKPIT  ACTIVATE 

•  PRErUGHT/POSTFLIGHT-NOGSE 

•  Blf  CIRCUIT  FAILURES  NOT  CAUSE 
OPERATIONAL  FAILURES 

•  INTEGRATE  GFE  BIT  CAPABILITIES 

•  BIT  PERFORMANCE 

-  DETECT  98%  OP  EQUIPMENT  FAILURES 

-  ISOLATE  9SX  OF  OETECTEO  TO  FAULTY  WRA 

-  99%  OF  FAILURE  INDICATIONS  CAUSEO  BY 
ACTUAL  FAILURES 


ELECTRONIC  EQUIPMENT  SPECIFICATIONS 
( P.S .  74-8700xx) 


•  JOENTIFY  FAILED  FUNCTIONS/MODES 

•  INITIATED  BIT  COMMANDED  BY  MC. 

MAY  INTERRUPT  OPERATION  BUT  NOT 
INTERFERE  WITH  ASSOCIATED  EQUIPMENT 

•  PcRIODIC  BIT  AT  90%  DETECTION 


•  01SPLAY  EQUIPMENT  MAY  UTILIZE 
TEST  DISPLAYS 


•  MC  WILL  ASSIST  MULTIPLEX  TERMINAL  TESTS 

•  INTEGRATION  SIGNALS 

-  INPUT  ACTIVATE.  MOLD.  STOP 

-  OUTPUT-  IN  TEST.  FUNCTION  STATUS, 

WRA  FAILEO.  EQUIPMENT  FAILED. 
EQUIPMENT  READY,  TEST  COMPLETE 

•  IDENTIFY  BIT  TEST  TIME.  CPS,  THRESHOLDS. 
TIME  OELAYS 


•  BIT  DESIGN/DEVELOPMENT  DATA.  ANALYSIS 
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F/A-18A  AVIONICS  BUILT-IN-TEST 
OPERATING  CHARACTERISTICS 


Figure  6-8 

*  PERIODIC  BIT 

-  AUTOMATIC  MONITORING  WITHIN  SUBSYSTEM  AFT  ER  POWER  UP 

-  HIGH  CONFIDENCE  OF  SUBSYSTEM  OPERATING  STATUS  (90% 

FAILURE  DETECTION) 


•  INITIATED  BIT 

-  INDIVIDUAL  OR  SIMULTANEOUS  SUBSYSTEM  SELECTION  BY  OPERATOR 

-  MAXIMUM  FAILURE  DETECTION  AND  ISOLATION  CAPABILITY  TO 
REMOVABLE  AVIONICS  WRA  (98%  DETECTION) 

-  AVAILABLE  ON  GROUND  AND  WHERE  SAFETY  PERMITS  IN-FLIGHT 


•  MAINTENANCE  BIT 

-  CALLS  UP  SPECIAL  CALIBRATION  ROUTINES  AND  DISPLAYS  FOR  MORE 
IN  DEPTH  SUBSYSTEM  TESTS 

-  EXPANDS  FAULT  ISOLATION  TO  PERIPHERAL  AIRCRAFT  COMPONENTS 
(RELAY  PANELS,  FLIGHT  CONTROL  SWITCHES,  ARMAMENT 
CONTROLS.  ETC) 

-  AVAILABLE  ON  GROUND  ONLY  AND  REQUIRES  UNIQUE 
OPERATOR  PARTICIPATION 


F/A-18A  AND  TF/A-18A  BUILT-IN-TEST 
ASSURANCE  ELEMENTS 


Figure  6-13 
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Figure  6-15 
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F-18  BUILT-IN-TEST  6  FrBRUARY  1981 
AIRCRAFT  INTEGRATION  AND  DEVELOPMENT 

Figure  6-l6 


AVIONIC  COOES* 


NEED  FURTHER  INVESTIGATION 
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Figure  6-17 
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F/A-18A  RELIABILITY  SUMMARY 
THROUGH  3, GOO  HOURS  OF  TEST  EVALUATION  Figure  6-18 
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FLIGHT  HOURS 


•  FLEET  EXPERIENCE  F-4J  -  0  69  MFHBF,  A-7E  -  0  P5  MFH8F 

•  CUMULATIVE  TEST  EXPERIENCE  *  OPERATIONAL  TYPE  GROUND  RULES  -  2.5  HR 
MFHDF  (ALREADY  BETTER  THAN  A  7  AND  F-4J) 

•  F/A-18A  ACHIEVED  ITS  RSLIA05U1 Y  REQUIREMENTS  BY  COMPLETING  THE 

50  FLIGHT  (100  FLIGHT  HOURS'  RELIABILITY  DEMONSTRATION  WITH  AN  8.4  HR 
MFHDF  tf.7  REQUIRED) 


AN/AYK-1 4(V)  BUILT  IN  TEST 

Wei  Long  Chen 
CONTROL  DATA  CORPORATION 

Mr.  Chen  is  a  Senior 
Diagnostics  Engineer  with 
Control  Data  Corporation 

HIGHLIGHTS  OF  THE  PRESENTATION 

The  BIT  development  sequence  for  the  AN/AYK-l4(V)  is  shown 
in  Figure  7-1.  AN/AYK-14  BIT  is  implemented  in  firmware,  as 
shown  in  Figure  7-2,  using  the  CDC  480  minoprocessor .  BIT 
performance  capabilities  are  shown  in  Figure  7-3,  including 
the  fault  isolation  within  the  5-0  milliseconds.  The  BIT 
indicator  stays  if  power  is  removed  before  indication  is 
recorded.  Fault  isolation  requirements  in  the  Navy  specifica¬ 
tion  are  86  percent  to  one  card  and  93  percent  to  two  cards. 

No  fault  detection  requirements  were  specified.  The  MTTR  and 
MTBF  requirements  specified  were  not  included  in  the  BIT 
requirements . 

Central  Processing  Unit,  Input  Output  Processors,  and 
Extended  Arithmetic  Unit  performance  capabilities  are  shown  in 
Figures  7-A,  7-5,  and  7-6. 

Design  verification  was  accomplished  by  a  preliminary 
qualification  test  including  algorithm  evaluation,  code  review, 
a  function  test  and  a  system  test  and  by  a  formal  qualification 
test  incorporating  customer  concurrence  that  all  requirement 
specifications  have  been  met.  The  BIT  Design  Review  Board  con¬ 
sisted  of  representatives  from  Government  and  from  the  areas 
of  firmware,  diagnostic,  and  hardware  design  as  well  as  quality 
assurance  and  reliability  and  maintainability.  Algorithm 
evaluation  was  accomplished  by  review  of  design  flow  charts, 
verification  that  the  program  design  addressed  the  fault 
detection  and  isolation  specification,  and  finally  the  genera¬ 
tion  of  a  report.  Code  review  was  accomplished  by  verification 
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that  the  code  is  error-free ,  that  it  complies  with  the  stand¬ 
ards  and  conventions,  that  it  implements  the  approved  design 
specs,  and  finally  the  generation  of  a  report.  Microcode 
programs  were  checked  by  other  microcode  programmers.  The  BIT 
code  is  contained  within  a  512  32-bit  micro-memory. 

Function  tests  were  performed  as  shown  in  Figure  7-7. 
Faults  were  selected  by  CDC.  Software  diagnostics  were  used 
as  a  backup  to  firmware  BIT  for  the  system  test  fault  list. 
System  cest  was  performed  as  shown  in  Figure  7-8.  Formal 
qualification  tests  were  performed  as  shown  in  Figure  7-9- 
The  government  selected  331  faults  and  witnessed  these  tests. 
BIT  firmware  has  been  accepted  by  the  government. 


112 


AN/AYK-14(V)  BUILT-IN-TEST 
DEVELOPMENT  SEQUENCE 


•  NAVY  REQUIREMENTS 

•  CONTROL  DATA  PROPOSAL 

•  CONTRACT 

•  PROGRAM  PERFORMANCE  SPECIFICATION 

•  INTERFACE  DESIGN  SPECIFICATION 

•  PROGRAM  DESIGN  SPECIFICATION 

•  ALGORITHM  EVALUATION 

•  IMPLEMENTATION 

•  CODE  REVIEW 

•  FUNCTION  TEST 

•  SYSTEM  INTEGRATION  TEST 

•  RELEASE  TO  CONFIGURATION  MANAGEMENT 
CONTROL 

•  FORMAL  QUALIFICATIONS  TEST  Figure  7-1 

•  BASELINE 


AN/AYK~14(V)  COMPUTER  SYSTEM 
BUiLT-IN-TEST 

•  BUILT-IN-TEST  IS  PART  OFAN/AYK-14(V)  FIRMWARE 


•  BUILT-IN-TEST  FIRMWARE  RESIDES  IN 
MICROMEMORY 


Figure  7-2  •  COMPUTER  ELEMENTS  WITH  BUILT-IN-TEST 


-  CENTRAL  PROCESSING  UNIT  (CPU) 

-  INPUT/OUTPUT  PROCESSOR  (IOP) 

-  EXTENDED  ARITHMETIC  UNIT  (EAU) 
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BIT  PERFORMANCE  SPECIFICATIONS 


OVERALL  PERFORMANCE 

—  INITIATED  ON  POWER-UP,  MASTER  CLEAR, 
INITIAL  PROGRAM  LOAD  (IPL) 


—  EXECUTES  IN  LESS  THAN  5.0  MILLISECONDS 


—  SETS  BIT  INDICATOR  WHEN  ERROR 
ENCOUNTERED 


Figure  7-3 


—  PROVIDES  STATUS/ISOLATION  CODE  WHEN 
ERROR  IDENTIFIED 


Figure  7-4 


CPU  BIT  PERFORMANCE  SPECIFICATION 

•  INITIATED  AS  A  RESULT  OF 

-  SYSTEM  POWER-UP 

-  HARDWARE  MASTER  CLEAR 

-  CONTROL  CONSOLE  MASTER  CLEAR 

-  INITIAL  PROGRAM  LOAD  (IPL) 

•  TEST  SCOPE 

-  MICROSEQUENCE 

-  ARITHMETIC  LOGICAL  UNIT  (ALU)  REGISTERS, 
U,  K,  AND  I  RFGISTERS 

-  ALU  LOGIC  NETWORK 

-  FILE  AND  C-FILE 

-  MEMORY  CONTROL  MODULE  (MCM)  PAGE  FILE 

-  CPU  BUS  AND  I/O  BUS 

-  EVENT  MONITOR 

-  MEMORY  PARITY 

•  SETS  BIT  INDICATOR  WHEN  ERROR  DETECTED,  AND 

-  IF  COMPUTER  SUPPORT  EQUIPMENT  (CSE)  IS 
CONNECTED,  FIRMWARE  HANGS  AND  AN  ERROR 
IDENTIFICATION  CODE  IS  SENT  TO  CSE  FOR 
DISPLAY. 

-  IF  NO  CSE  IS  CONNECTED,  BIT  RESTARTS. 

•  STARTS  THE  SYSTEM  INITIALIZATION  PROCESS  IF 
BIT  SUCCESSFULLY  COMPLETES. 

•  iN  A  DUAL  PROCESSOR  CONFIGURATION,  CHECKS 
IF  IOP  BIT  RAN  TO  SUCCESSFUL  COMPLETION. 
FIRMWARE  HANGS,  IF  IOP  FAILS  ITS  BIT. 
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IOP  BIT  PERFORMANCE  SPECIFICATION 

•  INITIATED  AS  A  RESULT  OF 

-  SYSTEM  POWER-UP 

-  MASTER  CLEAR 

•  TEST  SCOPE 

-  MICROSEQUENCE 

-  ALU  REGISTERS,  U,  K,  AND  I  REGISTERS 

-  ALU  LOGIC 

-  FILE 

-  I/O  BUS 

-  EVENT  MONITOR 

-  MEMORY  INTERFACE  Figure  7-S 

•  SETS  BIT  INDICATOR  WHEN  ERROR  DETECTED,  AND 

-  IF  CSE  IS  CONNECTED,  FIRMWARE  HANGS  AND 
AN  ERROR  IDENTIFICATION  CODE  IS  SENT  TO 
CSE  FOR  DISPLAY. 

-  IF  NO  CSE  IS  CONNECTED,  BIT  RESTARTS. 

•  INFORMS  CPU  OF  IOP  TEST  STATUS  IN  A  DUAL 
PROCESSOR  CONFIGURATION. 

•  STARTS  IOP  INITIALIZATION  PROCESS  IF  BIT  IS 
SUCCESSFULLY  COMPLETED. 

EAU  BIT  PERFORMANCE  SPECIFICATION 


Figure  7-6 


•  INITIATED  BY  CPU  BIT 

•  TEST  SCOPE 

-  ARITHMETIC  LOGICAL  UNIT  (ALU) 

—  PROGRAMMABLE  READ  ONLY  MEMORY  (PROM) 
CONSTANTS  CHECKSUM 

—  EXPONENT  ALU  LOGIC 

-  CONDITIONAL  REGISTER  AND  BRANCH  LOGIC 

—  RF.GISTER  SHIFT  LOGIC 

-  REGISTER  MULTIPLY  LOG>C 


•  SETS  ERROR  FLAG  IN  STATUS  WORD,  AND  AN 
ERROR  IDENTIFICATION  CODE  IS  SENT  TO  CPU 

•  ON  EXIT  (NORMAL  OR  ABNORMAL)  FIRMWARE 
ENTERS  IDLE  LOOP  TO  WAIT  FURTHER 
PROCESSING. 
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FUNCTION  TEST 


•  TEST  REQUIREMENTS 

-  COMPUTER  SUPPORT  EQUIPMENT  IS  REQUIRED 

TO  CONTROL  BIT  EXECUTION.  Mgure  l-l 

-  RELIABILITY  AND  MAINTENANCE  GROUP 
PROVIDES  A  LIST  OF  FAULTS  THAT  INCLUDES 
AT  LEAST  ONE  FAULT  TO  EXERCISE  EACH 
ERROR  STOP  IN  BIT  FIRMWARE. 

•  TEST  PROCESS 

-  HARDWARE  DESIGN  GROUP  APPROVES  THE 
FAULTS. 

—  FAULTS  ARE  INSERTED  ON  1C  CHIP  LEADS. 

—  VERIFIES  THAT  THE  DETECTED  FAULTS  ARE 
ISOLATED  CORRECTLY. 

-  EACH  UNDETECTED  FAULT  IS  ANALYZED  TO 
DETERMINE  ITS  RELEVANCE  TO  FUNCTIONAL 
PERFORMANCE. 

—  CORRECTS  BIT  FORMWARE  TO  PROVIDE 
DETECTION  OF  THE  RELEVANT  FAULTS. 

—  GENERATES  REPORT. 

»  THE  UNDETECTED  FAULTS  RELEVANT  TO 
FUNCTIONAL  PERFORMANCE  ARE  ADDED  TO  THE 
SYSTEM  TEST  FAULT  LIST. 

SYSTEM  TEST 

•  TEST  REQUIREMENTS 

—  COMPUTER  SUPPORT  EQUIPMENT  IS 
Figure  7-6  REQUIRED  TO  CONTROL  BIT  AND 

DIAGNOSTICS  SOFTWARE  EXECUTION. 

—  RELIABILITY  AND  MAINTENANCE  GROUP 
PROVIDES  A  LIST  OF  FAULTS. 

*  TEST  PROCESS 

-  FAULTS  ARE  INSERTED  INTO  1C  CHIP  LEADS. 

-  BIT  SUPPLEMENTED  BY  THE  DIAGNOSTICS 
SOFTWARE  IS  RUN. 

-  EACH  UNDETECTED  FAULT  IS  ANALYZED  TO 
DETERMINE  ITS  RELEVANCE  TO  THE  SYSTEM 
PERFORMANCE. 

-  CORRECTS  DIAGNOSTICS  PROGRAMS,  BIT  OR 
DIAGNOSTICS  SOFTWARE,  TO  PROVIDE  THE 
DETECTION  OF  THE  RELEVANT  FAULT. 

-  GENERATES  REPORT. 
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FORMAL  QUALIFICATION  TEST 


TEST  REQUIREMENTS 

-  COMPUTER  SUPPORT  EQUIPMENT  IS 
REQUIRED  TO  CONTROL  BiY  AND 
DIAGNOSTICS  SOFTWARE  EXECUTION. 

-  THREE  DIFFERENT  AN/AYK-14(V) 

CONFIGURATIONS  ARE  REQUIRED  TO  Figure  7-9 

UNDERGO  THIS  TEST. 

-  GOVERNMENT  PROVIDES  A  LIST  OF  FAULTS. 

-  THIS  TEST  MUST  BE  WITNESSED  BY 
GOVERNMENT  REPRESENTATIVES. 

TEST  PROCESS 

-  FAULTS  ARE  INSERTED. 

-  ON  EACH  CONFIGURATION,  BIT 
SUPPLEMENTED  BY  ITS  DIAGNOSTICS 
SOFTWARE  IS  RUN. 

-  ALL  FAULTS  THAT  ARE  NOT  DETECTED  OR 
ISOLATED  CORRECTLY  ARE  ANALYZED  FOR 
THEIR  LEGITIMACY. 

-  GENERATES  REPORT. 


AN/ALG- ! 26B 

DESIGNING  AND  VALIDATING  BIT 


Ken  Wilson 

MAINTENANCE  TECHNOLOGY,  INC. 

Mr.  Wilson  is  President 
of  Maintenance  Technology, 
Inc. 

HIGHLIGHTS  OF  THE  PRESENTATION 

The  requirements  evolution  of  the  AM/ALG-126B  has  derived 
from  recent  experience  with  the  AN/ALQ-126  and  the  AN/ALR-&5, 
and  fleet  availability  experience.  RISE  (Readiness  Improvement 
Scales  Evaluation)  of  the  ALQ-126  and  the  ALR-^5  provided  high 
level  visibility  indicating  tne  requirement  for  improvements. 
Both  were  specified  by  AR-10  criteria  tc  99  percent  fault 
detection  and  98  percent  fault  isolation.  With  ECM  equipment, 
the  only  performance  verification  is  provided  by  BIT  or  by  ETE, 
which  makes  3IT  very  important.  A  3^  percent  CMD  rate  or  re¬ 
moved  SRAs  at  "I"  level  was  observed  in  operation  of  the  ALQ- 
126.  In  1978,  all  boxes  were  removed  from  the  aircraft  and  put 
through  a  mod  program;  BIT  was  the  first  test  performed  on  the 
removed  units.  In  the  ALQ-126  mod  program  (Delta-Mod),  ^6  per¬ 
cent  of  the  units  removed  had  "hard"  failures  by  I-Ievel  tests, 
although  the  BIT  had  showed  them  to  be  good.  In  the  Colt--'!5 
mod  program — the  ALR-A5  update--55  percent  of  the  boxes  actually 
had  hard  failures  that  the  BIT  showed  to  be  good.  In  both 
cases,  validity  of  the  hard  failures  was  established  by  the  I- 
level  tests. 

It  appears  that  false  alarm  problems  early  in  the  programs 
may  have  resulted  in  the  spread  of  tolerances  in  the  BIT,  reduc¬ 
ing  its  effectiveness.  In  certain  RF  areas  where  BIT  was  not 
effective,  reliance  was  placed  on  external  test  equipment. 

For  the  ALQ-126,  I-levtl  bench  testing  initially  resulted 
.in  an  overall  field  MTTR  of  approximately  10  hours.  That  was 
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subsequently  reduced  to  6  hours,  but  never  achieved  the  labor¬ 
atory  demonstrated  MTTR  of  2  hours. 

The  parameters  specified  for  the  AN/ALQ-126B  are  shown  in 
Figure  3-1.  MTTR  was  specified  as  32  minutes  at  0-level  and  2 
hours  at  I-level.  The  same  AR-10  values  were  imposed  as  with 
the  ALQ-126.  I-level  testing  was  required  to  use  the  same  logi 
and  circuitry  as  the  BIT. 

To  ensure  timely  validation  of  the  ALQ-1265  BIT,  a  design 
review  team  was  established.  This -team  was  made  up  of  an 
experienced  systems  engineer,  dedicated  subsystems  engineers, 
a  test  equipment  engineer  and  a  field  engineer.  In-process 
validation  of  the  AN/ALQ-I26B  is  shown  in  Figure  3-2. 

The  joint  verification  process  of  the  AN/ALQ-26B  is  shown 
in  Figure  8-3,  using  each  requirement  of  the  specification.  No 
specific  cutoff  point  (e.g.,  10  percent)  for  BIT  was  estab¬ 
lished.  The  maintainability  demonstration  has  not  been  com¬ 
pleted  since  the  equipment  is  presently  in  T&E . 

Figure  8-4  provides  an  example  of  ALQ-126B  BIT  paper  veri¬ 
fication  as  related  to  the  specification  requirements. 

BIT  effectively  is  presented  in  Figure  8-5,  showing  an 
overall  BIT  effectivity  of  96.5  percent.  Comparisons  between 
the  ALQ-126  and  the  ALQ-126B  are  shown  in  Figure  8-6.  The  most 
significant  improvements  are  shown  in  the  increased  number  of 
fault  indications  (38  to  170),  the  decreased  percent  of  dedi¬ 
cated  BIT  hardware  (20  percent  to  2  percent),  and  the  increased 
percent  of  BIT  circuitry  tested  by  BIT  (1  to  99  percent).  It 
should  be  noted  that  the  ALQ-126  is  basically  an  analog  box 
and  the  ALQ-126B  is  primarily  a  digital  box.  Utilization  of 
BIT  logic  at  I-level  was  increased  from  0  to  95  percent. 
Interface  circuitry,  which  provides  for  future  wraparound  BIT, 
has  been  included  to  accommodate  inputs  from  other  interfacing 
units . 
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DISCUSSION  POINTS 


Functions  tested  are  identifed  by  specific  circuit- 
analysis  . 

BIT  effectivir.y  is  expressed  numerically. 

0-level  test  equipment  dependence  has  been  reduced. 

I-level  test  time  has  been  reduced,  using  the  same 
logic  from  0-level  to  I-level. 

False  removal  rate  is  projected  to  be  less  than  one 
percent . 


•  Cost  for  the  dedicated  BIT  engineering  is  approxi¬ 
mately  five  percent  of  the  non-recurring  costs.  An 
anticipated  reduction  in  production  testing  costs  will 
more  than  offset  the  development  investment.  In  addi¬ 
tion,  the  0-level  and  I-level  non-recurring  and  recurr¬ 
ing  test  equinment  costs  have  been  reduced  by  use  of 
BIT. 


•  There  is  initial  design  opposition  to  BIT  that  may  be 
overcome  by  successful  use  of  BIT  in  the  debug  process. 

•  Management  (government  and  contractor)  must  be  edu¬ 
cated  early  in  the  program  and  informed  of  the  signi¬ 
ficant  cost  benefits  achievable  by  incorporation  of 
BIT  early  in  the  design  phase. 

•  Defining  a  BIT  FOM  should  include  the  number  of  func¬ 
tions  tested,  the  amount  of  BIT  required  to  test  the 
functions,  the  reliability  of  the  BIT  circuitry  and 
the  accuracy  of  the  BIT  measurement. 

•  If  a  parameter  is  sufficiently  important  to  be  speci¬ 
fied  for  BIT  monitoring,  the  tolerances  for  3IT  moni¬ 
toring  should  also  be  specified. 

•  Partitioning  of  equipment  by  function  must  be  con¬ 
sidered  in  terms  of  BIT  operation. 
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AIM  /  ALQ-126B 

REQUIREMENTS  EVOLUTION 


•  SPECIFIED  PARAMETERS 

®  BIT  ACCURACY  (WITHIN  10%  OF  SPEC  VALUES) 

•  MTTR 

•  AR  10  {%  FAULT  DETECTION/ISOLATION) 

9  GOALS 

•  "0"  LEVEL  TEST  EQUIPMENT 

•  ELIMINATE 

•  T  LEVEL  TEST  PHILOSOPHY 

•  MINIMIZE  BENCH  LOADING 

Figure  8-1 


AN  /  ALQ-126B 
IN-PROCESS  VALIDATION 


•  CONTRACTOR  -  PROGRAM  MANAGEMENT 
IMPLEMENTATION 

•  PERSONNEL  SELECTION 

•  SYSTEM  ENGINEER 

•  SUB  SYSTEM  ENGINEERS 

•  HARDWARE 

•  SOFTWARE 

•  TEST  EQUIPMENT 

•  TRAINING 

•  DESIGN  REVIEW  PROCESS 

•  RESOURCE  ALLOCATIONS  _ .  0  , 

Figure  8-2 
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AN  /  ALQ-126B 
JOINT  VERIFICATION 


•  SPECIFICATION  ITEM  REVIEW 

•  BIT  CONTRIBUTION  TO  FAILURE  RATE 
V  BIT  EFFECTIVITY 

•  ”0"  LEVEL  TEST  EQUIPMENT  REQUIREMENTS 

•  "I"  LEVEL  BENCH  TIMES 

•  MAINTAINABILITY  DEMONSTRATION 

Figure  8-3 
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AN  /  ALQ-126B 

BIT  EFFECTIVITY  DERIVATION 


PREDICTED  FAILURE  RATE  DISTRIBUTION 


SELF  TEST  IDENTIFICATION  OF  FAILURES 
WEIGHED  BY  FAILURE  RATE  DISTRIBUTION 


Figure  8-5 


AN  /  ALQ-126B 
VERIFICATION 


COMPARISON  TO  AN/ALQ  I26 

126 

12GB 

NUMBER  FUNCTIONAL  TESTS  (FAULT  DETECTION) 

3 

7 

MAJOR  TEST  CASES  (RF  POWER,  SENSITIVITY,  ETC) 

5 

11 

SPECIFIC  TESTS  (RF  POWER  SATURATION  AND  MIN  SIGNAL) 

NUMBER  FAULT  INDICATIONS  (RF  FIRST,  SECOND  AMP  OUTPUT 

31 

53 

AMP) 

38 

170 

%  BIT  DEDICATED  HARDWARE 

m 

2% 

%  BIT  TESTED  BY  BIT 

<i% 

99% 

ANALYTICAL  DIAGNOSTICS  ("1"  LEVEL  UTILIZATION  OF  BIT  LOGIC) 

0 

95% 

BIT  PROVISIONS  FOR  INTERFACE  CIRCUITRY 

0 

80% 

Figure 
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AN/ S  PS  -  67  RADAR  BIT 

Mel  Nunn,  NOSC 
John  Rogers,  NAC 


Mr.  Nunn  is  the  head  of  the 
Test  Technology  Office  at 
NOSC.  Mr.  Rogers  is  a  test 
equipment  engineer  at  NAC.. 

HIGHLIGHTS  OF  THE  PRESENTATION 

The  AN/SPS-67  radar  is  a  Class  A  surface  search  radar 
which  replaces  the  AN/SPS-10.  It  was  developed  by  NORDEN, 
a  division  of  United  Technologies,  under  NAVSEA  contract.  The 
radar  completed  operational  evaluation  in  October-November  1980. 
It  originated  from  an  exploratory  development  program  that 
showed  that  life  cycle  costs  (I.CC)  of  Wavy  radar  systems  could 
be  reduced  by  50  percent,  using  the  technology  of  1971.  This 
program  was  called  "2175,"  "two  to  one  in  1975."  The  AN/SPS- 
67  radar  was  one  of  the  four  projects  in  the  2175  program.  At 
the  time  of  exploratory  development  there  were  fifty-plus 
radars  all  performing  the  same  surface  search  function.  Analy¬ 
ses  showed  that  the  functions  of  all  these  radars  could  be 
performed  by  two  radars  which  would  be  ninety-five  percent 
alike.  The  radars  were  partitioned  functionally,  using  46 
types  of  Navy  standard  electronic  modules  (SEMs'  nany  of  which 
had  already  been  developed  by  NAC;  approximately  29  SEMs  had  to 
be  developed  specifically  for  this  program.  BIT  and  design  for 
testability  had  been  specified  as  a  requirement  during  the  6.1 
and  6.2  phases,  and  implemented  in  the  6.3  phase.  In  the  fleet 
today  there  are  4  to  5  million  SEMs  in  over  150  types  of 
equipment.  Testability  (as  in  reliability,  maintainability, 
and  availability)  requires  a  firm  technology  base  to  support 
the  systems  in  which  it  is  incorporated. 

NAC  developed  the  exploratory  development  model  which  was 
delivered  and  qualified  prior  to  going  out  on  contract  to 
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Norden.  It  consisted  of  six  units,  including  the  dummy  load 
and  remote  control  unit.  The  major  unios  were  the  video 
processor  and  the  receiver  transmitter. 

BIT  requirements  for  the  AN/SP5-67,  as  specified  in 
SHIPS-R-5763A,  are  that  it  shall  locate  95  percent  of  failures 
to  the  four  SEMs  in  either  the  video  processor  or  the  receiver 
-transmitter;  that  it  shall  have  at  least  two  operating  modes 
(normal,  operational  and  interactive,  test);  and  that;  it  have 
automatic  self-test.  Other  specified  BIT  requirements  are 
shown  in  Figure  9-1.  Automatic  self-test  includes  testing  of 
27^  test  points  every  20  seconds.  Double  fault  testing  also 
is  used.  After  double  verification  of  the  fault,  the  BIT 
further  checks  the  fault  an  additional  16  times.  If  iu  still 
exists  at  that  time,  a  hard  fault  is  declared  and  an  alert  is 
displayed  to  the  operator  and  the  fault  identification  infor¬ 
mation  is  recorded  on  the  maintenance  panel.  Figure  9-2  iden¬ 
tifies  the  requirements  specified  for  the  interactive  BIT. 

The  operator  can  select  a  test  point  at  random  and  step  through 
test  points  one-at-a-time  around  the  selected  point.  Figure 
9-3  indicates  the  specified  testability  requirements  of  which 
BIT  is  a  subset.  Figure  9-1)  indicates  the  specified  maintain¬ 
ability  requirements. 

Maintainability  program  characteristics  are  shown  in 
Figure  9-5.  NAVBEA  Detachment,  Norfolk,  appointed  one  person 
to  follow  the  program  throughout  with  regard  to  its  maintain¬ 
ability,  thereby  providing  continuity  to  the  program.  Char¬ 
acteristics  specified  for  maintenance  personnel  are  that  they 
hold  an  ET3  maintenance  rating,  be  a  graduate  of  an  applicable 
Class  "C"  school,  and  have  nine  months  related  experience.  A 
Navy  ET3  is  equivalent  to  an  Air  Force  E*i .  "C"  school  runs 

from  two  to  six  weeks  on  a  particular  piece  of  gear  (the  SPS- 
67  required  two  weeks).  The  specification  of  qualifications 
of  the  maintenance  personnel  results  primarily  in  the  level 


of  writing  of  the  maintenance  manuals.  It  does  not  appear  to 
drive  the  equipment  design  over  and  above  the  specified  MTTR. 

Results  of  BIT  showing  isolation  to  an  average  of  four 
SEM  ambiguity  level  is  shown  in  Figure  9-6,  with  BIT  compris¬ 
ing  20  percent  of  the  total  circuitry.  A  more  intensive  BIT 
is  being  developed  for  the  digital  section,  whereby  a  known 
signature  is  introduced  into  the  front  end  and  later  read  out 
after  processing  and  compared  with  the  known  desired 
signatures . 

Verification  of  BIT  was  performed  through  a  factory  main¬ 
tainability  demonstration  conducted  by  a  Navy  ET3  personnel 
using  actual  clock  time.  A  total  of  57  tasks  were  selected. 
Factory  tests  showed  an  MTTR  of  30.3  minutes.  Maintainability 
tests  resulted  in  an  MTTR  of  22  minutes.  The  evaluation  of 
radar  reliability  estimated  a  3,000  hour  MTBF.  In  actual 
tests,  the  radar  demonstrated  2,895  nours  with  one  failure. 


DISCUSSION  POINTS 

•  The  Advanced  Technology  Strategy  Team  is  a  Navy 
strategy  team  consisting  of  representatives  from 
every  Navy  laboratory  and  most  of  the  field  activi¬ 
ties,  ?'esulting  in  a  total  of  20-plus  organizations. 
The  team  meets  on  a  quarterly  basis,  planning  and 
implementing  the  basic  research  programs  in  the  Navy. 
Test  technology  is  being  considered  in  the  6.1,  6.2, 
and  6.3  funding  categories,  especially  those  things 
which  are  considered  critical  to  the  Navy.  One  of 
the  documents  that  the  team  has  generated  is  the 
"BIT  Design  Guide."  It  includes  a  "strawman"  spec 
for  specifying  BIT,  and  BIT-related  definitions.  A 
false  alarm  is  considered  a  failure  as  defined  in 
this  guide.  The  guide  is  available,  on  request, 
through  N0SC. 

•  Testability  and  BIT  requirements  should  prevail 
throughout  the  specification  of  a  system,  using 
previous  experience  with  similar  systems  to  determine 
where  BIT  requirements  should  be  made  more  specific 
(e.g.,  VSWR  on  a  duplexed.  Design  reviews  should 

be  used  to  refine  decisions  on  implementation  of  BIT. 


•  MV/SC  has  developed  a  two  day  training  course  termed 
"Design  for  Testability"  which  has  recently  been 
presented  at  the  Navy  Postgraduate  School.  The 
course  will  be  offeree:  around  the  country  to  govern¬ 
ment  and  industry.  It  includes  testability  needs  and 
details  on  techniques  to  make  designs  testable.  ,Lt 
also  includes  analog,  digital  and  hybrid  circuits. 

It  deals  with  both  system  level  and  board  level  test¬ 
ability.  Measurements  of  achieved  testability  capa¬ 
bilities  are  also  included.  The  schedule  for  offering 
the  course  has  not  been  established.  (Notification 
will  be  sent  to  all  BIT  workshop  attendees.  Other 
details  on  the  course  are  available  from  Bill  Keiner 
at  the  NSWC.) 

•  A  Test  Technology  Library  has  been  established  at 
NOSC  including  information  on  such  areas  as  electro¬ 
optics,  fiberoptics,  bubble  memories,  microorocessor- 
based  boards,  etc.  The  data  base  for  the  design  for 
testability  has  been  provided  to  the  library  through 
Bill  Keiner  at  the  NSWC. 

•  A  "corporate  memory"  has  been  established  by  the  Navy, 
having  a  focal  point  of  one  or  more  individuals  at 
each  laboratory  and  fleet  center  (e.g.,  NEC, 

Charleston  Engineering  Center,  and  the  Fleet  Training 
Equipment  Center,  Orlando).  Quarterly  meetings  are 
held  among  members  with  assignments  for  program 
reviews,  etc.  For  basic  research  (6.]),  contact  is 
maintained  with  various  university  professors  as  part 
of  the  "corporate  memory",  encouraging  them  to  incor¬ 
porate  design  for  testability  into  the  curriculum  to 
address  long-term  problems.  However,  short-term  and 
medium-term  problems  cannot  be  handled  by  this 
approach. 
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SPECIFIED  BIT 


Figure  9-1 


o  FAILURE  SHALL  NOT  DEGRADE  SYSTEM  PERFORMANCE 

o  SHALL  DISPLAY  LOCATIONS  OF  FAULT  ON  INDICATORS 

o  SHALL  ALERT  OPERATOR  OF  MARGINAL  OPERATION 

o  SHALL  BE  CAPABLE  OF  MEASURING  AND  DISPLAYING 
SPECIFIED  FUNCTIONS 

SPECIFIED  INTERACTIVE  TEST  BIT  Figure  9-2 

o  SHALL  NOT  REQUIRE  EXTENSIVE  OPERATOR  INTERFACE 
o  SHALL  ALLOW  FOR  SELECTION  OF  SENSOR  TEST  POINTS 
o  SHALL  ALLOW  SEQUENTIAL  STEPPING  OF  TEST  EVENTS 


SPECIFIED  TESTABILITY  CHARACTERISTICS  Mg^e  9-3 


o  MODULATOR  -  TRANSMITTER  SHALL  HAVE  FAULT  DETECTION 
SENSORS  FOR  BIT  MONITORING 

o  DIRECTIONAL  COUPLER  SHALL  CONTAIN  PROVISIONS  FOR 
BIT  AND  EXTERNAL  TEST  EQUIPMENT 

o  VIDEO  OUTPUTS  FROM  VIDEO  DISTRIBUTION  AMPLIFIERS 
SHALL  BE  AVAILABLE  FOR  BIT 
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SPECIFIED  MAINTAINABILITY  CHARACTERISTICS 


pigure  9-^ 


o  MTTR  SHALL  NOT  EXCEED  .5  HOUR 

o  MAX  TTR  SHALL  NOT  EXCEED  1,5  HOURS  AT  THE 
35th  PERCENTILE 


SPECIFIED  MAINTAINABILITY  CHARACTERISTICS  Figur-  9-9 


o  MAINTAINABILITY  PROGRAM  PER  MIL-STD-A70 

o  MAINTAINABILITY  PREDICTION  PER  MIL-HDBK-A72 

o  MAINTAINABILITY  DESIGN  REVIEW 

o  DESIGN  AND  CONSTRUCTION  SHALL  PERMIT  RAPID  ASSEMBLY 
AND  DISASSEMBLY  FOR  EASE  OF  MAINTENANCE 
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RESULTS 


Figure  9- 6 


o  AVERAGE  AMBIGUITY  OF  FOUR  SEH 

-  ONE  ANALOG 

-  FIVE  TO  SEVEN  DIGITAL 

o  APPROXIMATELY  20%  OF  TOTAL  CIRCUITRY  IS  BIT 
o  NQ  "0"  LEVEL  TEST  EQUIPMENT  REQUIRED 
o  NO  FALSE  ALARMS  DURING  OP/TECH  EVALUATION 
o  m  TEST  POINTS 

o  SIGNATURE  ANALYSIS  CURRENTLY  BEING  DEVELOPED 
FOR  DIGITAL  SECTION 


THE  U.S.  NAVY'S  AEGIS 
WEAPONS  SYSTEM-ORTS 

Howard  Boardman 
Bob  Wood 

RCA-GOVERNKENT  SYSTEMS  DIVISION 

Mr.  Boardman  is  the  manager 
of  AEGIS  Combat  systems 
readiness  at  RCA.  Mr.  Wood 
is  the  manager  of  ORTS 
integration  and  test  at  RCA. 

HIGHLIGHTS  OF  THE  PRESENTATION 

The  AEGIS  weapon  system  (Hark  7)  consists  of  a  weapons 
control  system,  command  and  decision  center,  a  radar  system 
(AN/SPY-1A),  an  operational  readiness  test  system,  standard 
missile,  guided  missile  launching  system,  and  a  fire  control 
system.  The  key  performance  factors  of  the  AEGIS  are  reaction 
time,  firepower,  ECM  and  environmental  resistance,  coverage, 
and  continuous  availability. 

Real  world  availability  is  described  in  Figure  10-1. 
Requirements  were  allocated  to  various  portions  of  the  system. 
The  system  design  process  is  shown  in  Figure  10-2  showing  the 
relationship  between  the  FMEA  and  the  availability  model  ana¬ 
lysis.  The  test  design  requirements  are  allocated  to  ORTS  and 
BIT  as  shown  in  Figure  10-3-  Figure  10-4  shows  the  AN/UYK-20 
central  ORTS  computer  and  its  data  base,  as  well  as  its  inter¬ 
faces  with  the  AEGIS  weapons  system.  Data  Acquisition  Con¬ 
verters  (DACs)  are  used  to  develop  the  interfaces  between  the 
equipment  and  the  data  bases.  Figure  10-5  lists  the  ORTS 
performance  requirements.  ORTS  essentially  has  to  detect  all 
faults.  The  fault  isolation  requirements  in  the  specification 
are  that  95  percent  of  the  faults  will  be  processed  either 
automatically  or  semi-automatically  to  reach  the  designation 
of  an  LRU,  without  additional  test  equipment.  A  functional 
view  of  ORTS/ AEGIS  is  shown  in  Figure  10-6.  The  circled 
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.items  are  considered  to  be  BIT  functions.  The  ORTS  system  is 
considered  a  macro-BITE  within  the  AEGIS  system. 


The  AN/SPY-1 A/ORTS  test  configuration  is  displayed  in 
Figure  10-7.  Of  the  181.2K  core  in  the  SPY-1  computer,  approx¬ 
imately  ten  percent  is  devoted  to  the  self-testing  function. 

A  total  of  4,500  test  points  are  examined  through  the  47  DACs, 
including  178  on-line  detection  tests.  The  AN/UYK-20  ORTS 
computer  program  uses  about  52K  words  of  core  with  another  25K 
non-core  resident.  ORTS  f?les  are  shown,  indicating  the  rela¬ 
tive  sizing,  totalling  roughly  10  to  12  percent  of  the  tacti¬ 
cal  system.  The  tactical  disk  storage  capability  is  20  M 
words.  ORTS  uses  2  to  2.5  words  total. 

Figure  10-8  shows  a  typical  maintenance  sequence.  The 
initial  fault  detection  comes  from  the  system  AN/UYK-7  com¬ 
puters.  Evaluation  is  performed  by  the  AN/UYK-20  ORTS  com¬ 
puter.  Fault  detection  and  identification  outputs  of  ORTS 
are  shown. 

AEGIS/ORTS  was  installed  in  1973  on  the  USS  Norton  Sound 
for  evaluation  and  test  firings.  Prior  to  this,  RCA  built  a 
replica  of  the  Norton  Sound  deck  house,  called  the  Land  Based 
Test  Site  ( LBTS ) .  The  same  Navy  crew  operating  the  LBTS  was 
sent  to  the  Norton  Sound  to  conduct  the  tests.  Presently, 

RCA  has  another  facility  at  Moorestown  representing  the  AEGIS 
cruiser  deckhouse  that  has  been  operating  for  two  years.  The 
Norton  Sound  crew  is  presently  operating,  testing  and  debugg¬ 
ing  this  facility  and  will  go  to  the  first  AEGIS  Cruiser,  the 
USS  Ticonderoga.  Tests  are  controlled  by  the  Combat  System 
Maintenance  Central  (CSMC)  including  test  conduct,  personnel 
schedule,  deferred  maintenance,  etc.,  providing  central 
control  for  all  maintenance. 

A  third  facility  has  been  constructed  at  Moorestown  for 
test  and  checkout  of  the  production  AEGIS,  using  ORTS  in  the 
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system  verification  process.  It  can  test  two  AEGIS  production 
systems  at  a  time. 

The  verification  approach  and  0R7S  requirements  are  shown 
in  Figure  10-9.  "One-on-one"  represents  the  cabinet -to-DAC 
interface  which  was  developed  at  the  RCA  Burlington  facility 
and  at  Raytheon.  The  173  on-line  detection  tests  may  sample 
10  to  20  test  points  at  a  time  to  determine  the  operation  of 
a  function.  Some  fault  isolation  tests  may  run  from  20  to  300 
pages  of  flow  charts  for  a  single  diagnostic  test  on  a  single 
cabinet.  The  test  plan  must  include  all  ORTS  resting  to  deter¬ 
mine  the  percentage  of  equipment  tested  (including  ORTS).  A 
good  data  collection  system  is  required  to  correlate  ORTS 
design  with  actual  failures.  A  "Trouble  and  Failure  Report” 
has  been  developed  by  the  Navy  and  RCA  to  provide  supplementary 
failure  reporting  for  AEGIS.  The  CSMC  aboard  ship  also  in¬ 
cludes  a  representative  of  RCA. 

Tools  and  methods  used  in  design  and  evaluation  of  ORTS 
are  shown  in  Figure  10-10.  Special  data  base  utility  programs 
allow  the  modification  of  parameter  values  and  tolerances  to 
accommodate  design  changes.  The  Utility  to  Save  and  Restore 
the  Disk  (USAR)  utility  program  permits  taking  the  data  base 
from  the  disk  and  putting  it  on  magnetic,  tape.  It  can  later 
be  loaded  on  disk,  if  required.  This  capability  permits  test 
coordination  among  different  test  teams  at  different  locations. 
The  on-line  radar  test  puts  35  32-bit  words  into  the  ORTS  I/O 
buffer,  telling  the  SPY-1  radar  to  perform  certain  actions. 

The  return  from  this  radar  is  27  32-bit  words.  This  sequence 
is  termed  a  "test  dwell."  It  is  on-line  and  is  interleaved 
with  the  operational  dwells  on  a  non-interruptive  basis.  In- 
terruptive  radar  tests  are  held  to  a  minimum. 


Operational  Test  Definition  and  Criteria  are  shown  in 
Figure  ±0-11.  Fault  ID  numbers  are  on  the  order  of  15,000 
to  20,000  different  numbers. 

Things  to  verify  are  shown  in  Figure  10-12.  Operational 
performance  is  achieved  by  utilizing  the  same  people  that  are 
going  to  operate  the  system  in  the  fleet. 

Integration  and  test  phasing  of  ORTS  are  shown  in  Figure 
10-13.  I&CO  tests  are  qualitative,  while  O&P  and  performance 
tests  are  quantitative.  The  diagram  represents  a  time  period 
of  2*J  to  26  months. 

The  status  of  ORTS  integration  and  checkout  is  shown  in 
Figure  10-1*1.  It  is  expected  to  be  completed  by  the  end  of 
the  year.  Conclusions  reached  as  a  result  of  the  Aegis/ORTS 
Program  are  shown  in  Figure  10-15. 

DISCUSSION  POINTS 

•  A  Test  Requirements  and  Analysis  (TRA)  group  was 
established  to  receive  test  requirements  from  the 
various  design  engineers  prior  to  incorporating  them 
in  the  test  plan  for  fault  detection  and  fault 
isolation. 

®  Two  test  teams  were  used  for  a  period  of  two  years; 
each  team  (of  three  to  six  persons)  was  at  the  site 
four  hours  per  day  to  complete  testing  of  the  EDM 
system. 
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HIGHLIGHTS  OF  THE  PRESENTATION 

Some  of  the  problem  areas  discovered  by  RADC  in  certain 
exploratory  development  (6.2  funding  category)  studies  were: 
unnecessary  removals,  high  false  alarm  rates,  and  low  fault 
isolation  and  fault  detection  percentages.  A  30  to  ^0  percent 
RTOK  rate  was  found  quite  universal.  RADC  awarded  a  contract 
to  Hughes  Aircraft  to  perform  a  field  survey  to  investigate 
and  verify  the  RTOK  problem,  document  RTOK  rates  and  determine 
a  means  to  minimize  removals.  Nine  aircraft  types,  six  equip¬ 
ments,  twelve  bases  and  two  repair  facilities  were  included  in 
the  survey.  Definitions  used  are  shown  in  Table  1  of  the 
attachment  (excerpted  from  the  study  report). 

The  main  results  of  the  study  are  indicated  in  Figure 
11-1.  Primarily  it  was  found  that  maintenance  personnel  in 
the  field  had  to  resort  to  "unseemly"  troubleshooting  prac¬ 
tices.  Since  they  sometimes  don't  know  where  the  problem  is, 
they  resort  to  indiscriminate  removals  of  boxes.  In  addition, 
there  is  a  "horizontal  testability"  problem  where  a  unit  will 
test  '■good"  at  one  AIS  and  "bad"  at  another  AIS  at  the  same 
facility.  Easy  accessibility  as  required  for  most  LRUs  en¬ 
courages  indiscriminate  removals.  In  some  cases,  adjacent 
boxes  require  one  to  bo  removed  in  order  to  obtain  access  to 
the  latches  on  the  other  box.  In  this  case,  both  boxes  are 
frequently  sent  back  to  the  shop.  Depot  practices  of  cleanup 
prior  to  testing  and  repair  may  result  in  the  removal  of  the 
fault  (e.g.  dirty  connector)  during  cleanup  and  the  consequent 
RTOK.  Inconsistencies  were  found  in  filling  out  the  3^9 


forms  ar.d  transferring  information  to  the  350  tags  among  bases. 
All  maintenance  actions  are  pilot  driven  and  sometimes  pilots 
report  failures  that  don't  exist. 

The  study  results  also  showed  a  mean  value  of  38  percent 
for  unnecessary  removal  rates  which  ranged  from  4  percent  to 
89  percent  for  different  equipment  LRUs.  Sometimes,  unnecessary 
adjustments  are  made  at  the  I-level  shop  in  order  to  avoid  a 
maintenance  event  being  termed  an  unnecessary  removal,  possibly 
resulting  in  the  numbers  shown  being  conservative. 

Study  of  the  false  alarm  problem  included  the  equipments 
shown  in  Figure  11-2  for  the  data  base.  Categories  of  false 
alarms  included  Category  I,  a  system  failure  but  an  isolation 
to  the  wrong  box,  and  Category  II,  the  classic  false  alarm 
where  there  is  no  failure  in  the  system  but  BIT  says  there  is. 
False  alarm  rates  for  the  F-15  radar  systems  and  the  F-l4  radar 
systems,  respectively,  were:  F-15  AN/APG-63  Category  I — 38 
percent,  Category  II  —  22  percent;  P-14  AN/AWG-9  Category  I  — 

28  percent.  Category  II — 53  percent,  where  the  percentage  is 
the  percent  of  fault  indications. 

Faulc  isolation  problems  on  the  S-3A  aircraft  are  shown 
In  Figur®  11-3,  based  on  a  study  by  Lockheed.  Actual  fault 
isolation  percentages  to  the  SRU  level  all  fall  far  short  of 
design  percentages.  This  indicates  that  maintenancer  personnel 
are  not  being  given  enough  assistance.  Severe  turnaround  time 
requirements  sometimes  force  the  removal  of  three  or  four 
boxes  instead  of  isolating  to  one  box.  Removal  and  checkout 
of  one  box  at  a  time  requires  more  aircraft  down  time. 

Results  of  an  ASP  study  of  several  avionic  equipments  are 
shown  in  Figure  11-4;  the  results  are  summarized  for  the  four 
less  successful  equipments.  These  results  show  a  direct 
correlation  between  the  number  of  fault  detections  and  the 
number  of  false  alarms  declared.  In  addition  to  the  equip¬ 
ment's  shown,  the  AN/ARC-164  showed  the  same  relationship. 
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In  this  case,  the  BIT  defection  capability  in  the  ARC-164  was 
reduced  and  the  false  alarm  rate  also  decreased.  Mo  loss  in 
operational  capability  was  observed  as  a  result  of  this  de¬ 
crease.  Overspecification  (e.g.  9°  percent  detection,  one 
percent  false  alarm  rate)  is  a  contradiction  that  does  not 
appear  possible  to  achieve  at  the  present  time. 

Objectives  of  the  RADC  Figure  of  Merit  (FOM)  Study  are 
shown  in  Figure  11-5.  Specific  BIT  objectives  are  categorized 
in  Figure  11-6.  A  survey  of  industry  resulted  in  the  FOM  list 
shown  in  Figure  11-7.  These  FOMs  are  associated  with  fault 
detection  capability  in  Figure  11-8  and  with  fault  isolation 
caoability  in  Figure  11-9-  The  most  significant  physical  and 
operational  characteristics  affecting  BIT  are  shown  in  Figure 
11-10. 

Analysis  and  demonstration  techniques  applicable  to  each 
FOM  are  shown  in  Table  2  of  the  Attachment;  some  of  these 
techniques  are  summarized  in  Figures  11-11,  11-12,  and  11-13. 
RADC  has  a  joint  program  with  the  Naval  Electronics  Systems 
Command  (Washington,  D.C.)  to  develop  improved  methods  of 
maintainability  prediction  to  supplement  or  replace  some  of 
the  procedures  in  MIL-HDBK-472  addressing  BIT  capabilities. 

It  is  presently  being  circulated  for  comments.  BIT  suitability 
factors  were  included  in  the  industry  survey  and  addressed  the 
factors  shown  in  Figure  11-14.  They  were  ranked  in  accordance 
with  their  suitability  to  meet  the  specification  objectives 
shown  in  Figure  11-15.  The  BIT  FOM  selection  process  used  in 
the  study  is  shown  in  Figure  11-16  for  each  candidate  FOM. 
Results  of  selection  of  BIT  FOMs  are  summarized  in  Table  6  of 
the  Attachment. 

The  maintenance  concept  objective  to  improve  flight  line 
efficiency  by  removing  only  the  one  failed  LRU  and  to  improve 
I-level  efficiency  by  removing  only  the  one  failed  SRU .  Cal¬ 
culations  were  performed  to  determine  how  many  removals  can  be 
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expected  for  each  system  failure.  The  methodology  used  in 
these  calculations  is  shown  in  Figure  11-17.  In  this  figure, 

X  is  a  measure  of  fault  isolation  capability,  P(MA)  is  the 
probability  ^f  missed  assignment  (erroneous  fault  indication), 
and  HR  is  removal  rate  (which  is  dependent  upon  maintenance 
policy) . 

A  typical  example  of  ENR  (expected  number  of  removals) 
calculations  is  provided  in  Figure  11-18,  showing  1^1  boxes 
removed  for  every  100  failures  in  an  instance  when  single  unit 
removal  maintenance  policies  were  used.  Under  conditions  of 
a  group  removal  policy,  the  numbeh  becomes  177  boxes  removed 
for  every  ICO  failures. 

Requirements  for  logistic  specifications  are  shown  in 
Figure  11-19.  Logistic  specification  FOM  relationships  to 
costs  are  shown  in  Figure  11-20.  ENR  is  relatable  to  Mean 
Time  Between  Removal  (MTBR) .  It  is  measurabl-. ,  traceable  and 
relates  directly  to  life  cycle  costs. 

FCM  conclusions  and  recommendations  are  shown  in  Figure 
11-21. 

The  methodology  to  apply  a  MIL-STD-471  demonstration  is 
shown  in  Figure  11-22.  Figure  11-23  shows  the  deficiencies 
in  this  methdology  today.  For  example,  if  the  SPO  is  not  ade¬ 
quately  monitoring  the  selection  of  the  failure  population, 
FMEAs  are  sometimes  not  performed,  hazardous  failures  are 
not  identified,  and  failure  samples  may  not  be  randomly 
selected.  In  addition,  the  MIL-STD-^71  demonstration  is  per¬ 
formed  in  a  sterile  environment  and  does  not  reflect  field 
conditions . 
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UNNECESSARY  RFHQVALS 


FINDINGS  TQ  DATE 

FACTORS  THAT  CONTRIBUTE  TO  UNNECESSARY  REMOVALS 
o  BUILT- IN-TEST 

o  TEST  EQUIPMENT 

o  TECHNICAL  ORDERS 

o  MAINTENANCE  PROCEDURES 

o  TRAINING 

o  ACCESSIBILITY 

o  MAINTENANCE  CONCEPTS 

o  DEPOT  PRACTICES 

o  DATA  COLLECTION  Fisur>e  n-i 


o  RADC  CONTRACTUAL  EFFORT  WITH  HUGHES  AIRCRAFT  SUPPORT  SYSTEMS  GROUP 
o  ANALYSIS  OF  BIT  FALSE  ALARMS  CONDITIONS 

1.  FUNDAMENTAL  CAUSES  AND  FREQUENCY 

2.  DESIGN  GUIDELINES 

3.  PREDICTION  FACTORS 
o  DATA  BASE 

1.  AN/APG-63  FIRE  CONTROL  RADAR  (F-15) 

2.  AN/AWG-9  RADAR  &  WEAPONS  CONTROL  (F-l'l) 

Figure  11-2 

3.  TPQ-37  ARTILLERY  LOCATING  RADAR  (ARMY) 
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BIT  FOM  SPECIFICATION  &  DEMONSTRATION 


mnm. 

o  BIT/EXTERNAL  TEST  FIGURES  OF  MERIT  AND  DEMONSTRATION  TECHNIQUE  (RADC-TR-79- 
309), 

o  SURVEY  &  INVESTIGATE  CURRENT  MEASURES  AND  FIGURES  OF  MERIT  (FOM)  USED  TO 
SPECIFY  BUILT-IN-TEST  (BIT)  AND  EXTERNAL  TESTER  (ETE)  ADEQUACY. 

o  DETERMINE  METHODS  OF  MEASUREMENT  AND  DEMONSTRATION  FOR  THE  FOMs. 

o  PROVIDE  GUIDANCE  AS  TO  HOW  TO  SPECIFY  THESE  FOMs. 

o  PROVIDE  GUIDANCE  FOR  INTEGRATION  OF  BIT/ETE  REQUIREMENTS,  ANALYSIS,  AND 
DEMONSTRATION  INTO  CURRENT  MAINTAINABILITY  PROGRAM  PLANS. 

Figure  11-5 
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Figure  11-6 


1.  FRACTION  OF  FAULTS  DETECTED  (FFD) 

2.  FRACTION  OF  FALSE  ALARHS  (FFA). 

3.  FRACTION  OF  FALSE  STATUS  INDICATIONS  (FFS1). 

I.  MEAN  FAULT  DETECTION  TIME  (TfD>. 

5.  MEAN  BIT  RUHIN6  1IHE  (Tb). 

6.  FREOUENCY  OF  BIT  EXECUTIONS  (Fg). 

7.  TEST  THOROUGHNESS  (TT). 

8.  FAULT  ISOLATION  RESOLUTION  (FIR(D). 

9.  FRACTION  OF  FAULTS  ISOLATED  (FFI). 

10.  MEAN  FAULT  ISOLATION  TIME  <TFI>. 

II.  MAINTENANCE  PERSONNEL  SKILL  LEVEL  (HPSL). 

12.  BIT  MAINTAINABILITY  (lITTRg/g). 

13.  BIT  RELIABILITY  (MTBFg/E>. 
i9.  BIT  AV1AILABILITY  (Ag/E>. 
lb.  EQUIPMENT  AVAILABILITY  (A). 

16.  EQUIPMENT  MAINTAINABILITY  (MTTR). 

17.  FRACTION  OF  FALSE  PULLS  (FEP) 

18.  FRACTION  OF  ERRONEOUS  FAULT  ISOLATIONS  (FEF1) 


Figure  11-7 
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0  COST 
0  WEIGHT 
0  VOLUME 

o  Complexity 


o  MTBFB/E 
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Figure  11-10 


BIT  FOM  SPECIFICATION  a  DEMONSTRATION 


ffi  /MLISiS-imiiilM  demonstration  irfiimniir 

FFD  CAN  BE  ANALYZED  BY  A  RATIO  OF  OCCURRENCE  RATES 
(E.G.,  FAILURE  RATE) 

FFA  CAN  BE  ANALYZED  BY  A  RATIO  OF  OCCURRENCE  RATES 
(E.G.,  FAILURE  RATE) 

FFSI  CAN  BE  ANALYZED  BY  A  RATIO  OF  OCCURRENCE  RATES  CAN  BE  VERIFIED  BY  FIELD  DATA 

(E.G.,  FAILURE  RATE)  COLLECTION  ONLY 

TfO  CAN  BE  ANALYZED  BY  A  METHOD  SIMILAR  TO  HIL-IIDBK-  CAN  BE  VERIFIED  BY  DIRECT  TIME 

'i/Z  PROCEDURE  2  OR  RADC-TR-78-1G9,  A  FAILURE  RATE  MEASUREMENT 

WEIGH  FED  AVERAGE  OF  TIHES  (TIHES  DETERMINED  THRU 
TIME  LINE  ANALYSIS) 

Figure  11-11 


CAN  BE  VERIFIED  BY  A  BINOMINAL 
DISTRIBUTION  OR  BY  FIELD  DATA 
COLLECTION 

CAN  BE  VERIFIED  BY  FIELD  DATA 
COLLECTION  ONLY 


157 


BIT  FOM  SPECIFICATION  &  DEMONSTRATION 


(FFD,  FEF1,  TT,  FFI) 

FIND  PROBABILITY  OF.  PASSING  A  TEST  AS  A  FUNCTION  OF  THE  TRUE  FOM  VALUE  FOR 
FIXED  n  AND  k. 

FIND  n  AND  K  WHERE  DESIGN  FOM  GOAL,  MINIMUM  FOM  ACCEPTABLE,  AND  CONSUMER  AND 
PRODUCERS  RISKS  ARE  GIVEN. 

FIND  DESIGN  FOM  GOAL,  MINIMUM  FOM  ACCEPTABLE,  AND  CONSUMER  AND  PRODUCERS  RISKS 
FOR  GIVEN  n  AND  K. 


WHERE  n  =  SAMPLE  SIZE 

K  -  PASS/FAIL  CRITERIA 

Figure  11-12 


BIT  FOM  SPECIFICATION  &  DEMONSTRATION 


MULTINOMIAL  DISTRIBUTION 


FIR(L) 


TYPICALLY  EXPRESSED  AS  MULTIPLE  REQUIREMENTS 
DEMONSTRATE 

1.  SPECIFY  MINIMUM  ACCEPTABLE  CRITERIA 

2.  SPECIFY  CONSUMER/PRODUCER  RISK 

5.  DERIVE  TEST  STATISTIC  (T)  AND  TEST  CRITERIA  (C> 


PRODUCER  PASSES  IF  TiC 


Figure  ll-i; 


PIT  FfiM  SELECT IMJRQ££SS 

SlUIABlUIL£CIfiR$ 


1.  UNIQUENESS 

2.  TRACK ABILITY 

3.  DEH0NSTRATAB1L1TY 

H,  TR  AN  Si_  AT  AB 1 L 1 TY 

5.  ambiguity 


6.  APPLICABILITY 


Figure  11-lA 


RlT  FQh  sllegiqilerqcess 
SPFC 1 F 1  CAT  1  ON  JjBJEClHES 

1.  CONTINUOUS  PERFORMANCE 

2.  ACCURATE  STATUS  REPORTING 

3.  MINIMUM  DOWNTIME 

n,  limited  spares 

$.  MINIMUM  SUPPORT  COSTS 

6.  SPECIFIC  SKILL  LEVELS 

7.  SYSTEM  LOCATION 

g,  BIT  INTEGRAL  TO  SYSTEM 

5,  ON-LINE  TESTING  LENGTH/PERIODICITY 

1U.  SOFTWARE  TESTING  LIMITATIONS 

11.  UNDETECTED  FAILURES 

12.  BIT  OPERABILITY/ACCURACY 

13.  PHYSICAL  AMOUNT  OF  BIT  Figure  11-15 
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FIGURES  OF  MERIT 


SPECIF1CAT10I 
OBJECTIVES 

Rj  -  RANKING  DEE  TO  SUITABILITY  FACTORS 
R2  -  RANKING  DUE  TO  SPECIFICATION  OBJECTIVE 
Ra  -  AVERAGE  RANKING 

Figure  11-16 


"MISASSIGNMcNTS 


"MISASSIGNMENTS 


EXPECTED  NUMBER  OF  REMOVALS 
ENR  =  SYSTEM  FAILURE 

EXAMPLE 

M  -  5  P(MA)  =  -05 

X  =  1.07  FAR  =  •  -05 

P(FD)  =  .90 

ENR  -•  1.41 

Figure  11-18 

I  (Mil  STIC  SPiCiEiOIiMS 
SPECIFIC  REQUIREMENTS 


1.  FAULT  DETECTION 

2.  FAULT  ISOLATION 

3.  ERRONEOUS  FAULf  INDICATIONS 


GENERAL  REQUIREMENTS 

1,  REMOVALS  PER  FAILURE 

2,  FALSE  PULL  RATE 

3,  MTTR 

l6l 


Figure  11-19 


inciSTIC  SPEC1F1CAIM 


Figurell-20 


CONCLUSI ONS/RECOHMENDAT l OHS 
CURRENT  FOHs  ARE  ADEQUATE  IF  PROPERLY  DEFINED 
ROMs  CAN  BE  GENERAL  OR  SPECIFIC 

f0«s  CM  BE  SPECIFIED  KITH  HIIIM  BfKI  «  «AINTWNABIL!T( 

PROGRAMS 

1,  MIL- STD- 470 

2.  MIL- STD- VI 

Figure  11-21 
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RANDOMLY  SELECT 
FAILURE  POPULATION 
(SPO  MONITORED) 


PERFORM  FMEA 


IDENTIFY  HAZARDOUS 
&  SECONDARY  FAILURES 


RANDOMLY  SELECT 
FAILURE  SAMPLE 
FOR  DEMONSTRATION 


SPO  REPORT 
&  CORRECTIVE  ACTION 


RjpetfLf  SELECT 
FAIIURE  POPULATION 
(SPO  JJpurfORED) 


Figure  13-22 
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&  CORRECTIVE  ACTION 

FOR  DEMONSTRATION 

Figure  13 
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BUILT-IN-TEST  SPECIFICATION 
&  DEMONSTRATION  TECHNIQUES 

(ATTACHMENT) 


CAPT  DANIEL  GLEASON 
R&M  ENGINEERING  SECTION 
ROME  AIR  DEVELOPMENT  CENTER 
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BIT  WORKSHOP  PANELS 


Martin  Meth 
OSD,  MRA&L 

Mr.  Meth  is  the  Staff 
Engineer  for  OSD,  MRA&L. 

Three  panels  have  been  formed  for  the  purpose  of  discussing 
perceived  major  issues  involving  BIT.  Each  panel  is  asked  to 
identify  what  is  currently  being  done,  and  what  could  be  done 
to  resolve  some  of  the  identified  problems  in  the  near,  inter¬ 
mediate  and  long  term.  The  three  panels  address  the  problems: 
how  do  we  specify  BIT,  how  do  we  do  development  tests,  and  how 
do  we  do  operational  test. 

The  Requirenents  Panel  will  address  the  questions  of  how 
the  user,  the  operator,  and  the  program  people  define  a  set 
of  requirements  against  which  we  should  design  BIT  and  other 
diagnostics.  How  should  manpower  and  readiness  questions  be 
taken  into  account  (broadly  scoping  these  requirements,  in¬ 
cluding  the  resources  required  to  support  system  development). 

In  the  past,  poor  planning  (including  not  setting  aside  test 
articles)  nas  resulted  in  schedules  being  grossly  underesti¬ 
mated  It  appears  that  more  thinking  and  planning  should  be 
done  during  the  requirements  phase. 

The  Subsystem  Panel  will  address  the  questions  of  how  we 
specify  requirements  for  equipments  such  as  radars,  how  we  do 
the  development  testing,  and  the  design  veri fiction  to  deter¬ 
mine  that  what  we  specify  is  actually  built  into  the  equipments. 
In  other  wo^ds,  what  should  we  tell  the  contractor,  what  should 
we  expect  back  from  him  in  terms  of  response,  and  what  we  ex¬ 
pect  in  terms  of  design  validation. 
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The  System  Panel  will  address  the  questions  of  how  we 
specify  requirements  on  the  system  given  that  we  have  a  lot  of 
equipments  comprising  the  system,  how  we  do  system  design  veri¬ 
fication,  and  how  we  operationally  test  the  system  to  see  that 
the  requirements  have  been  met. 

Panel  members  have  been  arbitrarily  selected.  The  three 
panels  will  present  (tomorrow  morning)  half-hour  reports  in¬ 
cluding  problem  statements,  observations  and  recommendations; 
from  this  we  may  arrive  at  a  consensus  of  whether  there  are 
answers  to  certain  problems  and,  if  there  are,  what  those 
answers  are  most  likely  to  be.  Recommendations  for  present 
alternatives  and  for  future  research  should  be  included. 

Each  panel  has  been  provided  with  one  copy  of  all  the  vu- 
graphs,  for  reference,  as  well  as  a  copy  of  all  the  homework 
comments  turned  in  by  the  workshop  members.  Panels  will  meet 
for  approximately  two  hours  this  afternoon. 

The  question  has  been  raised  as  to  whether  another  BIT 
Workshop  would  be  appropriate  which  would  deal  with  specific 
methods  and  techniques  for  implementation  of  BIT  (since  this 
present  workshop  deals  with  certain  management  aspects  of  BIT 
such  as  specification,  verification  and  testing  of  BIT). 
Recommendations  from  the  three  panels  are  requested.  Although 
the  scope  of  such  a  workshop  may  be  extended  to  include  other 
BIT  subjects,  specific  BIT  Implementation  would  probably  be  the 
primary,  and  possibly  the  only,  subject  of  the  workshop. 

Many  programs  underway  today  appear  to  be  repeating  many 
of  the  mistakes  of  past  programs.  It  is  important  that  the 
messages  of  this  workshop  get  to  the  people  running  these 
programs  to  avoid  compounding  these  mistakes.  Panel  recommen¬ 
dations  are  requested  in  this  area. 

DISCUSSION  POINTS 

«  It  is  apparent  that  many  people  in  the  BIT  community 
are  not  aware  of  the  existence  of  pertinent  documents 
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such  as  some  RADC  Bit  evaluation  reports.  Those 
interested  in  being  put  on  the  RADC  distribution  list 
should  contact  Dan  Gleason  or  Tony  Coppola. 

An  Automatic  Testing  Newsletter  is  published  by  NAVMAT 
(George  Neumann).  It  can  be  used  to  advise  the  BIT 
community  of  the  existence  of  various  reports.  (At 
the  present  time,  many  of  the  people  in  the  BIT 
community  do  not  receive  that  newsletter.) 

Other  newsletters  exist  that  might  contain  information 
of  benefit  to  the  BIT  community  such  as  APSC,  IEEE 
Reliability  Society,  AAE,  and  ASQC. 

The  tri-service  JLC  annual  meeting  is  being  held  in 
September,  1981  in  San  Diego.  Participation  by  mem¬ 
bers  of  the  BIT  community  would  be  appropriate  to 
further  exchange  information. 


INSTRUCTIONS  TO  THE  GROUPS 


Summarize  for  each  issue  or  observation  the  major  problems 
and  proposed  solutions.  The  following  should  be  considered  as 
the  issues  are  addressed: 

(1)  Does  the  group  agree  that  the  statements  of  issues  and 
observations  are  within  the  defined  scope?  If  not,  add  to  or 
change,  as  appropriate. 

(2)  Characterize  the  issues  and  observations  in  terms  of 
degree  of  criticality,  anticipated  ease  or  difficulty  of 
resolution,  and  priority  for  resolution. 

(3)  Identify  whether  the  problems  associated  with  each 
issue  and  observation  are  technical  or  managerial. 

(4)  In  describing  the  solutions,  discuss  the  adequacy  of 
what  is  currently  being  done  to  resolve  the  problems?  what 
changes  should  be  immediately  implemented.  What  could  be 
accomplished  in  the  next  two  years,  five  years  and  how  should 
these  changes  be  implemented.  Also  discuss  what  specific  parts 
of  the  acquisition  process  (e.g.,  specifications,  procedures) 
need  to  be  changed  and  suggest  changes. 

(5)  Define  the  EIT  terms  used  in  the  group  discussion. 

(6)  Identity  research  areas  that  should  be  pursued  to 
improve  the  specification,  test  and  evaluation  of  BIT, 


'll 
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GROUP  #1  REQUIREMENTS 


Wessel  (Leader) 


Ger.et 

Nunn 

Walters 

Shafer 

Griffith 


GROUP  #2 


Lindeni 

Faggard 

Cappola 

Jenkins 

Towsen 

Kauffman 


Drummond 

Bragg 

Charles 

Dudek 

Shebell 


SUBSYSTEMS  Keiner  (Leader) 


Schumacher 

Meyer 

Victor 

Chen 

Harrison 


GROUP  #3  SYSTEMS 


Gleason  (Leader) 


REQUIREMENTS  AND  EFFECTIVENESS  (GROUP  U) 


Scope :  Statement  of  Program  BIT  requirements;  relationship 
of  BIT  requirements  to  maintenance  capability  as  well  as  broader 
operational  needs;  evaluation  of  tradeoffs  between  BIT  require¬ 
ments  and  other  related  program  requirements — test  equipment., 
personnel  skills,  spares. 

Issues/Observations :  The  relationship  between  BIT  performance 
and  operational  i  eeds  for  logistics  and  manpower  requirements  are 
not  well  understood. 

1.  In  what  terms  should  program  BIT  requirements  be  expressed — 
broad  maintenance  diagnostic  terms?  BIT  contract  specifications  or 
both?  How  should  the  BIT  requirements  be  related  to  operational 
needs?  (a)  Operational  needs  encompass  indications  to  the  system 
operator  that  tie  system  is  operating  satisfactorily  and  (b)  the 
capability  to  f.'nd  system  failures  and  return  to  operational  status 
to  meet  system  vailability  or  usage  requirements. 

2.  What  should  be  the  process  for  formulating  the  program 
BIT  requirement  ? 

a.  Whc  should  be  responsible  for  developing  these 
BIT  requirements? 

b.  Whc  n  should  the  BIT  requirements  be  developed? 

c.  Wh  t  tradeoffs  need  to  be  considered  in  developing 
BIT  requirement  and  related  personnel  skills,  test  equipment, 
and  spares  requ  remen ts. 


177 


Issues/Observations :  Inadequate  resources  and  unrealistic 
schedules  are  being  proposed  for  development  of  BIT. 

1.  What  should  be  the  process  to  determine  if  the  program 
3IT  requirements  are  realistic  or  achievable? 

2.  How  should  program  funding  and  development  schedules 
be  estimated  for  BIT  development?  What  factors  should  be 
considered — test  facilities,  test  articles,  etc. 


S UBSYSTEM  AND  SPECIFICATION  TESTING  ( GROUP  # 2 ) 


Scope :  Equipment  which  is  part  of  a  major  system;  new  or 
existing  design;  BIT  requirements  for  contracts;  validation  of 
BIT  for  contractual  compliance;  evaluation  of  BIT  performance 
on  maintenance  capability  and  operational  needs. 

Issues/Observations :  The  results  of  contractor  BIT  demon¬ 
strations  (using  fault  insertions)  do  not  match  BIT  performance 
in  the  field. 

1.  How  should  BIT  requirements  be  specified  in  a  subsystem 
contract?  What  specific  terms  should  be  used?  Should  the  speci¬ 
fications  include  both  quantitative  measures  (i.e.,  percent 
faults  detected)  and  design  guidelines  (i.e.,  partitioning  of 
functions) ? 

2.  Should  different  BIT  terms  be  specified  in  contracts 
for  different  types  of  equipment?  Different  mechanizations? 

Issues/Observations :  Analysis  of  BIT  design  efforts  are 

not  providing  data  to  determine  if  BIT  specifications  can  be  met. 

1.  What  type  of  design  data  and  analysis  should  the  sub¬ 
system  contractor  use  to  manage  the  3IT  design/development 
efforts?  What  data  should  be  provided  to  and  by  the  system 
contractor? 

2.  How  should  the  contractor  demonstrate  compliance  with 
BIT  contractual  requirements  prior  to  subsystem  delivery? 

Should  BIT  demonstrations  include  both  analysis  and  testing? 

What  should  be  the  specific  "deliverables"  by  the  contractor. 
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What  are  the  limitations  of  contractor  BIT  demonstrations?  What 
test  areas  need  improvement?  (Considerations — fault  selection 
approach  and  sample  size) . 

3.  What  other  tests  that  are  usually  pe ■ 'ormed  during 
subsystem  development  can  be  used  in  the  determination  of  BIT 
contractual  compliance. 

4.  How  can  the  subsystem  contractor  get:  visibility  of 
subsystem  BIT  field  performance  at  the  level  of  detail  required 
to  find  and  fix  def iciencies? 

Issues/Observations :  Funding  and  schedule  allocated  for 

3IT  development  are  generally  not  adequate. 

1.  What  type  of  analysis  needs  to  be  completed  prior  to 
contracting  to  show  that  the  specified  BIT  performance  values 
are  feasible  and  achievable. 

2.  How  should  the  funding,  design  and  test  schedules,  test 
facilities  and  articles  be  estimated  for  the  BIT  subsystem 
development? 


180 


I 


SYSTEM  REQUIREMENTS  AND  TESTING  (GROUP  #3) 


Scope :  Systems  or  equipment  used  in  the  miltiary  operations; 
integration  of  subsystems  into  system;  specification  of  BIT 
requirements  in  contracts;  demonstration  of  contractual  compliance 
operational  test  and  evaluation  of  BIT. 

Issues/Observations :  Performance  of  BIT  in  the  field  is 
generally  much  lower  than  observed  in  testing  prior  to  production. 

1.  What  is  the  best  way  to  communicate  BIT  performance 
requirements  to  the  system  developer? 

2.  How  should  BIT  be  specified  in  contracts?  What  specific 
terms  should  be  used?  Should  the  specifications  include  both 
quantitative  measures  (i.e.,  percent  fault  detected)  and  design 
guidelines. 

3.  How  should  the  system  contractor  demonstrate  compliance 
with  the  BIT  specifications.  Should  compliance  be  demonstrated 
by  tests,  analysis  or  both. 

4.  What  should  be  the  operational  test  measures  of  BIT 
performance?  Kow  should  these  measures  be  related  to  BIT 
contract  requirements,  program  BIT  requirements? 

5.  What  are  the  areas  of  BIT  performance  that  cannot  bo 
evaluated  until  after  the  system  is  fielded? 

Issues/Observations ;  Current  approaches  to  BIT  system  design 
do  not  take  into  account  many  real  world  problems  as  evidenced 
by  high  levels  of  false  alarms,  undetected  failure,  retest  ok's 
and  ambiguities. 


1.  Should  the  BIT  contract  specifications  for  systems  also 
include  subsystem  specifications?  Should  the  subsystem  allocation 
consider  criticality  (mission  safety,  etc.)  feasibility  and/or 
operational  information  measures.  How  should  the  subsystem/ 
system  interface  requirements  be  specified? 

2.  What  type  of  design  data,  testing,  analysis  should  the 
system  contractor  use  to  manage  the  system  and  subsystem  BIT 
design  efforts?  What  data  and  analysis  should  be  provided  to 
the  program  office? 

Issues/Observations :  BIT  performance  in  the  field  has  not 
been  compatible  with  planned  personnel  skills,  test  equipment, 
and  other  logistics. 

1.  How  should  requirements  for  skill  level  of  personnel, 
test  equipment  and  spares  be  included  in  the  BIT  specifications? 

2.  To  what  extent  can  BIT  performance  measured  under 
operational  test  conditions  be  related  to  manpower  requirements, 
test  equipment  performance  and  spares  requirements? 

Issues/Observations :  Resources  and  schedule  allocated  for  BIT 

development  are  generally  not  adequate. 

1.  What  factors  need  to  be  considered  in  estimating  whether 
the  BIT  contract  requirements  are  realistic  and  achievable  and 
that  appropriate  resources  (funds,  test  assets,  etc.)  have  been 
budgeted. 

2.  What  ways  can  be  used  to  combine  contractor  demonstra¬ 
tion  of  BIT  with  field  demonstrations  to  develop  a  sense  of  how 
well  the  BIT  works. 
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BIT  WORKSHOP  PANEL  NO.  1  REPORT: 

REQUIREMENTS  AND  EFFECTIVENESS 

PANEL  CHAIRMAN:  LT .  COLONEL  JIM  WESSEL,  USAF 

Lt.  Colonel  Wessel  is  Chief 
of  the  Logistics  Assessment 
Procedures  Division  of 
AFTEC 

HIGHLIGHTS  OF  THE  PANEL  REPORT 

BIT  performance  vs  operational  needs  is  a  problem  which 
generally  is  not  well  understood--the  Panel  concensus  was  that 
this  is  a  true  statement.  In  fact,  the  operational  user  really 
does  not  have  and  should  not  specify  BIT  requirements  in  terms 
of  traditional  parameters,  such  as  percentage  of  PD  and  PI. 

The  operational  user's  requirements  should  be  in  the  form 
of  sets  of  constraints  consisting  of  such  operational  and  main¬ 
tenance  parameters  as  turnaround  time,  maximum  down  time,  man¬ 
power  levels,  skill  levels,  and  self  sufficiency  in  deployment. 
FD  and  FI  should  not  be  specified  by  the  user.  During  the 
process  of  system  definition,  the  optimum  "diagnostic"  system 
should  be  defined  within  the  user  constraints  consisting  of 
automatic  and  manual  diagnostic  capabilities  (Figure  13-D- 
This  system  consists  of  BIT,  people,  T.O.s,  and  test  equipment. 
The  contractor  and  the  procurer  must  then  define  the  required 
diagnostic  system  identifying  how  much  is  automatic,  how  much 
is  manual,  and  the  percentages  of  FD  and  FI. 

Figure  13-2  addresses  the  issue  of  the  relationship  of 
BIT  requirements  to  operational  needs.  It  is  shown  that  there 
are  two  basic  operational  needs  of  the  diagnostic  system:  fault 
detection  for  the  operator  and  fault  isolation  for  the  main- 
tainer.  The  operator  needs  fault  detection  to  answer  the 
questions:  what  have  I  got,  what  can  I  do,  what  don't  I  have, 
what  can't  I  do.  His  needs  for  information  are  different  from 
those  of  the  maintainer.  The  operator  does  not  need  to  know 
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which  LRU  is  bad.  He  needs  to  be  provided  with  mission-critical 
information.  The  maintenance  person  needs  fault  isolation  in 
order  to  know  which  LRU  is  bad  in  order  to  repair  it.  The 
operational  user  should  identify  for  the  producer/contractor 
upfront  what  the  information  needs  are  for  the  operator  and  for 
the  maintainer.  Normally  these  needs  are  different. 

Figure  13-3  addresses  the  issue  of  the  requirements  formu¬ 
lation  process.  The  Panel  consensus  was  that  it  is  an  inter¬ 
active  process  between  user  and  procurer  and  between  procurer 
and  contractor.  The  user  and  supporter  (depot)  should  define 
their  constraints;  this  should  be  done  between  Milestone  0  and 
Milestone  1  (preliminary  operational  concept).  The  diagnostics 
concept  should  be  fairly  well  firmed-up  by  Milestone  1,  but  not 
totally  defined  at  this  point.  The  extent  to  which  this  can  be 
done  by  Milestone  1  is  questionable  except  for  addressing  higher 
level  operational  constraints.  The  trade-offs  between  perform¬ 
ance  requirements  versus  skill  levels  and  between  test  equip¬ 
ment  versus  spares  was  not  addressed  in  great  detail.  The 
Panel  consensus  was  that  tradeoffs  were  required  in  these 
areas,  within  operational  and  support  constraints,  to  optimize 
the  operational  effectiveness  of  the  weapon  system.  Sometimes 
the  user  must  be  told  that  his  requirements  cannot  be  met  with¬ 
in  the  constraints  allowed.  Support  constraints  and  life  cycle 
costs  have  to  be  considered  in  context  with  performance  require¬ 
ments  when  tradeoffs  are  made.  The  procurer  must  make  it  clear 
to  the  contractor  how  requirements  are  specified  and  exactly 
what  they  mean  in  order  for  the  contractor  to  determine  how  he 
is  going  to  mechanize  the  system  in  terms  of  automatic  and 
manual  operation,  test  equipment,  skill  levels,  etc.  to  meet 
user  requirements.  The  key  premise  to  the  discussion  is  that 
the  diagnostic  system  must  provide  100  percent  FD  and  100  per¬ 
cent  FI,  although  it  is  not  all  automatic.  Tradeoffs  between 
lower  skill  levels  and  smarter  BIT,  as  well  as  tradeoffs 
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between  fewer  spares  and  more  automatic  test  equipment,  should 
be  reasonably  well  determined  by  Milestone  1.  What  is  imple¬ 
mented  in  the  equipment  itself  needs  to  be  determined  by  Mile¬ 
stone  2. 

Figure  13-^  addresses  the  issue  that  inadequate  resources 
and  unrealistic  schedules  are  being  proposed  for  system  develop¬ 
ment.  The  consensus  of  the  Panel  is  that  this  is  a  true  state¬ 
ment.  Determination  of  whether  the  BIT  requirements  are  real¬ 
istic  and  achievable  does  not  present  a  problem  if  development 
of  the  diagnostic  system  is  considered  a  systems  engineering 
discipline  as  related  to  all  other  ILS  elements.  If  that  is 
recognized,  development  of  the  diagnostic  system  with  its  BIT 
support  should  be  the  same  kind  of  process  as  required  to  get 
reliability  and  maintainability  into  the  system.  A  diagnostic 
development  program  is  required,  including  a  program  plan.  The 
system  may  include  the  imposition  of  ATE  interface  requirements 
such  as  with  the  MATE  concept.  The  BIT  program,  with  estab¬ 
lished  milestones,  assists  in  the  evaluation  of  the  realism  of 
forecast  BIT  achievements. 

Figure  13-5  addresses  the  issue  of  how  to  estimate  funding 
and  schedules.  This  issue  was  not  addressed  by  the  Prnel  be¬ 
cause  of  time  limitations.  My  opinion  is  that  if  the  diagnostic 
discipline  is  defined  and  implemented  as  ere  other  disciplines, 
funding  and  schedule  requirements  for  diagnostics  will  be  ad¬ 
dressable  as  are  those  in  the  other  development  disciplines. 

The  items  listed  in  Figure  13-5  are  those  which  should  be  con¬ 
sidered  in  this  area. 

DISCUSSION  POINTS 

•  By  not  recognizing  the  differences  between  operator 
FD  information  requirements  and  maintainer  FI  infor¬ 
mation  requirements  too  much  information  may  be  given 
to  the  operator  who  may  then  be  interpreting  it  in¬ 
correctly  and  possibly  forcing  more  maintenance  than 
would  otherwise  be  necessary.  On  the  other  hand,  in 
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seme  cases  too  little  information  may  be  provided  the 
operator  in  terms  of  "what  can  I  do"  relative  to  com¬ 
pletion  of  the  mission. 

•  By  Milestone  1,  higher  level  constraints  (e.g.  whether 
there  is  any  0-level  test  equipment)  should  be  re¬ 
solved,  permitting  the  contractor  to  determine  how 
he's  going  to  implement  that  requirement  in  the 
design. 

•  Differences  exist  between  how  the  contractor  evalu¬ 
ates  his  system  and  how  the  user  evaluates  it.  For 
example,  in  the  F-16,  the  100  percent  fault  detec¬ 
tion  seen  by  the  contractor  was  seen  as  83  percent  by 
the  user. 

•  Every  contractor  has  stated  that  he  needed  a  dedicated 
test  bed  or  dedicated  time  on  a  test  bed  and  a  long 
period  of  time  in  order  to  debug  and  mature  the  BIT. 

•  The  concept  of  BIT  flexibility  to  change  and  to  accom¬ 
modate  changes  and  unforeseen  events  is  important  in 
BIT  design.  This  flexibility  must  be  designed  up¬ 
front  to  accommodate  transitions  from  development 
through  production  phases. 

•  Schedule  requirements  to  mature  current  BIT  systems 
indicate  that  roughly  two  years  are  required  after  the 
system  is  fielded  to  mature  the  BIT.  This  requires  a 
maintenance  evaluation  team  dedicated  to  gathering 
data  on  how  well  the  diagnostics  functions  and  sorting 
out  where  the  problems  are  (hardware,  software,  teso 
equipment,  manuals,  etc.).  Prior  to  being  fielded, 
the  BIT  had  passed  some  military  demonstration 
requirements . 

•  No  attempt  has  been  made  to  document  the  cost  and 
schedule  requirements  necessary  to  mature  the  BIT  in 
existing  systems  for  use  in  projecting  requirements 
for  future  systems.  A  knowledge  base  has  not  been 
developed  in  this  area  to  determine  what  data  to 
collect . 

•  If  a  BIT  program  has  not  been  implemented  from  the 
very  beginning,  it  is  usually  impractical  and  cost 
prohibitive  to  do  so  after  the  fact. 

•  Design  for  testability  must  become  as  much  a  part  of 
the  design  discipline  as  design  for  performance 
presently  is.  Dedicated  test  articles  and  dedicated 
time  in  the  test  program  assist  in  attaining  this 
obj  ective . 

•  The  MATE  concept  (a  system  engineering  discipline),  if 
imposed  as  a  test  interface  in  a  system  such  as  the 


Multi-Role  Radar  (MRR),  will  enforce  a  diagnostics 
design  discipline.  If  it  is  not  imposed,  a  prolifera¬ 
tion  of  test  equipment  will  result, 

•  Cost  estimates  for  acquisition  and  test  of  supporta- 
bility  elements  of  the  system  are  estimated  at  20  per¬ 
cent  of  the  system  acquisition  costs.  BIT  costs  are 
included  in  this  amount. 

•  P-18  cost  estimates  for  BIT  are  seven  percent  of  the 
hardware  and  1 4  percent  of  the  software  costs.  Devel¬ 
opment  and  validation  of  the  design  is  an  interactive 
process . 

•  More  effective  utilization  of  testbeds  is  essential  to 
the  future  maturation  of  diagnostics. 

•  Limited  0-level  support  equipment  should  be  considered 
as  an  alternative  to  no  0-level  support  equipment  in 
some  cases. 
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ISSUE:  BIT  PERFORMANCE  VS  OPERATIONAL 
NEEDS  NOT  WELL  UNDERSTOOD 


PROBLEM:  TERMS  FOR  PROGRAM  BIT  REQUIREMENTS? 


Figure  13-1 


PROBLEM:  BIT  REQUIREFENTS  VS  OPERATIONAL  NEEDS 
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Figure  13-2 


PROBLEM*.  (EQUIIOENTS  FORMULATION  PROCESS? 


WHEN?  -  MILESTONE  1 

TRADE  OFFS?  BIT  ys.  SKjLLS  vs.  TEST  EQUIPMENT  vs  SPARES 

OPS  OPERATIONAL  EFFECTIVENESS  SUPPORT 
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Figure  13-3 


ISSUE*.  INADEQUATE  RESOURCES  &  UNREALISTIC  SQTELULES 
PROBLEM:  PROCESS  TO  DETERMINE  IF  BIT  RGMTS  REALISTIC 
&  ACHIEVABLE? 

ILS  ELEMENT 

SYSTEMS  ENGINEERING  DISCIPLINE 
PROGRAM  PLAN 


Figure  13-A 


PROBLEM:  HCW  TO  ESTIMATE  FUNDING  &  SCHEDULES? 
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Figure  13-5 
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BIT  WORKSHOP  PANEL  NO.  2  REPORT: 
SUBSYSTEM  BIT 


PANEL  CHAIRMAN:  BILL  KEINER 

Mr.  Keiner  is  the  head  of 
the  Testing  Technology 
Program  Office  at  the  NSWC 

HIGHLIGHTS  OF  THE  PANEL  DISCUSSION 

The  four  main  points  addressed  by  the  Subsystem  31?  panel 
were  BIT  design,  specification,  evaluation  and  management. 

Many  of  the  conclusions  of  this  panel  are  the  same  as  those  of 
the  System  Panel  (No.  3)  and  won't  be  repeated  here. 

BIT  design  recommendations  are  shown  in  Figure  1*1-1.  De¬ 
cent  alized  (e.g.,  subsystems)  BIf  is  recommended,  maintaining 
flexibility  to  adjust  thresholds  and  tolerances  'where  possible. 
BIT  processing  technology  ("smart  BIT")  should  be  developed 
further. 

BIT  specification  recommendations  are  shown  in  Figure  1*1-2. 
Numerical  BIT  requirements  should  not  be  included  in  a  MIL-STD; 
they  should  be  derived  for  each  system  on  an  individual  basis. 
Systems  that  include  multiple-application  subsystems  (e.g., 

GFE)  should  have  specification  values  for  R&M  that  take  into 
account  previous  experience  with  these  subsystems. 

BIT  evaluation  recommendations  are  shown  in  Figure  1*1-3. 
Early  design  analyses  of  BIT  are  essential  to  the  development 
process.  For  example,  in  the  case  of  VHSIC  elements,  BIT  must 
be  included  at  the  time  of  initial  circuit  design.  No  oppor¬ 
tunities  are  available  for  later  incorporation. 

Environmental  tests  should  be  combined  with  MIL-STD-*I71 
tests  tc  more  clearly  represent  the  operational  environment  and 
permit  reduction  of  the  post-development  debugging  time.  More 
should  be  done  to  assure  that  the  failure  types  demonstrated 
during  factor  acceptance  testing  represent  those  which  occur 
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in  the  field  (e.g.,  cable  problems).  Failure  data  relating  to 
BIT  performance  should  be  collected  during  all  types  of  testing 

BIT  management  recommendations  are  shown  in  Figure  14-4. 
BIT  must  take  its  proper  place  in  the  overall  integrated  diag¬ 
nostic  system. 


DISCUSSION  POINTS 

•  Adequacy  and  consistency  of  definitions  is  lacking, 
causing  many  communication  problems  in  the  BIT  com¬ 
munity;  this  problem  does  not  appear  to  be  adequately 
addressed  at  the  present  time.  The  BIT  Design  Guide 
includes  some  definitions,  other  definitions  are  in¬ 
cluded  in  FiADC  reports,  but  a  comprehensive  set  of 
definitions  is  not  available.  A  MIL-STD  1309C  (Navy) 
will  address  BIT  definitions. 

•  It  appears  that  a  single  focal  point  for  BIT  should 

be  established  within  system  and  subsystem  contractors 
facilities. 

•  Adequate  management  methods  and  motivation  do  not 
appear  to  exist  at  the  present  time  (within  government 
and  industry)  to  implement  guidelines  and  directive: 
relating  to  BIT  requirements. 

•  There  does  not  appear  to  be  a  viable  methodology  to 
predict  future  BIT  performance  based  on  field  experi¬ 
ence  with  similar  types  of  svstems  (e.g.,  F-18  and 
F-15). 

•  In  one  system,  the  BIT  was  very  good  and  was  used  in 
factory  acceptance  testing  and  resulted  in  a  saving 
of  two  percent  of  the  recurring  production  cost  be¬ 
cause  of  reduced  testing  costs.  Therefore,  in  some 
cases,  early  design  for  testability  can  actually  save 
front-end  acquisition  dollars,  rather  than  waiting 
for  the  payoff  in  long-term  life  cycle  costs.  How¬ 
ever,  data  are  generally  not  available  to  confirm 
this  saving.  These  type  of  data  should  be  collected 
to  give  proper  management  and  design  emphasis  to 
testability . 

«  It  is  more  efficient  so  use  different  types  of  testers 
during  the  design  checkout  phase  since  different  types 
of  failures  are  being  sought  (e.g.  design  errors)  than 
those  which  occur  during  service  use.  However,  the 
final  tests  in  factory  acceptance  should  be  done  with 
the  BIT  and  ATE  to  be  used  in  the  field. 
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•  Supportability  must  be  included  in  initial  design 
considerations  to  ensure  that  the  system  is  support¬ 
able  at  the  time  of  release  for  production.  In  the 
past,  the  supportability  of  systems  has  been  "shoved 
under  the  rug"  when  considering  the  decision  for 
release  for  production. 

•  The  airlines  are  beginning  to  establish  a  data  bank 
and  central  repository  on  BIT. 

•  The  JLC  Panel  on  Automatic  Testing  has  established 
focal  points  in  each  of  the  Services  for  testability 
tasks . 

•  A  "BIT  Community"  and  a  set  of  principles  do  not 
presently  exist  similar  to  the  "Reliability  Community" 
This  should  be  established  using  newsletters  as  a 
vehicle  for  communications .  This  BIT  Workshop  effort 
should  be  continued  to  help  organize  this  community. 


BIT  EVALUATION  RECOMMENDATIONS 


IMPLEMENT  BIT  AT  SUBSYSTEM  LEVEL 

KEEP  BIT  DESIGN  FLEXIBLE  TO  ALLOW  BIT 
MATURATION 

EMPLOY  INSTRUMENTATION 
TO  IDENTIFY  INTERMITTENTS 
TO  ASSIST  LATER  TESTING 

DEVELOP  BIT  PROCESSING  TECHNOLOGY 

CONSIDER  DIFFERING  NEEDS  OF  OPERATOR  AND  MAINTENANCE  MAN 

Figure  14-1 


BIT  SPECIFICATION  RECOMMENDATIONS 


BIT  PARAMETERS  TO  BE  SPECIFIED  SHOULD  BE  CHOSEN  FROM  A 
STANDARD  LIST  OF  WELL-DEFINED  PARAMETERS  WHICH  HAVE  WELL- 
DEFINED  VERIFICATION  METHODS 

NUMERICAL  BIT  REQUIREMENTS  MUST  BE  DERIVED  FROM  OPERATIONAL 
AND  LOGISTIC  REQUIREMENTS  THROUGH  TRADE-OFF  ANALYSIS 

SPECIFICATIONS  FOR  MULTIPLE-APPLICATION  SUBSYSTEMS  MUST 
BE  BASED  UPON  WORSE  CASE  HISTORICAL/PREDICTED  OPERATIONAL 
AND  LOGISTIC  REQUIREMENTS 

DRAFT  RFPs  MAY  BE  USED  TO  OBTAIN  FEEDBACK  ON  REASONABLENESS 
OF  REQUIREMENTS 

ALL  FAILURES  MUST  BE  ACCOUNTED  FOR,  TRADING  OFF  AUTOMATIC/ 
MANUAL  TEST,  BIT/ATE,  ETC, 


Figure  14-2 
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BIT  EVALUATION  RECOMMENDATIONS 


HOLD  BIT  DESIGN  REVIEWS.  REQUIRE  BIT  ANALYSIS  EARLY  ENOUGH 
TO  IMPACT  SYSTEM  DESIGN.  REQUIRE  FAVORABLE  BIT  PREDICTIONS 
BEFORE  PROCEEDING  WITH  DEVELOPMENT. 

USE  BIT  AND  TECHNICAL  MANUALS  AS  PART  OF  FACTORY  ACCEPTANCE 
AND  IN  TEST  AND  EVALUATION. 

DEMONSTRATE  BIT  USING  MIL- STD  471  PLUS  .  .  . 

MAKE  EXTENSIVE  USE  OF  I NSTRl MENTATION  TO  PERMIT  CHARACTER¬ 
IZATION  OF  FAILURES 

ESTABLISH  A  CLOSED  LOOP  DATA  SYSTEM  TO  MEASURE  BIT 
EFFECTIVENESS. 


Figure  14-3 


BIT  MANAGEMENT  RECOMMENDATIONS 


REQUIRE  AN  INTEGRATED  DISGNOSTIC  PROGRAM 

CONSIDER  DIAGNOSTICS  AS  A  SYSTEMS  ENGINEERING  DISCIPLINE 
DEVELOP  DISGNOSTIC  IN  PARALLEL  WITH  HARDWARE 
EMPHASIZE  VERTICAL  TESTABILITY 

PROVIDE  PROPER  VISIBILITY  AND  FUNDING  EARLY  IN  DEVELOPMENT 

ESTABLISH  CORPORATE  MEMORY 

CENTRAL  REPOSITORY  (TECHNIQUES/PERFORMANCE/COST  DATA) 
LESSONS  LEARNED 
SINGLE  POINT  OF  CONTACT 
DEFINE  BIT  "COMMUNITY" 

ESTABLISH  STANDARD  TERMINOLOGY 
PROVIDE  EDUCATION  FOR  MANAGERS 


Figure  14-4 


BIT  WORKSHOP  PANEL  NO.  3  REPORT : 
SYSTEM  BIT 


PANEL  CHAIRMAN:  CAPTAIN  DAN  GLEASON,  RADC 

Captain  Gleason  is  the 
Maintainability  Design 
Engineer  for  the  RADC 

HIGHLIGHTS  OF  THE  PANEL  REPORT 

Issues  and  observations  addressed  by  the  System  BIT  panel 
are  shown  in  Figure  15-1,  through  the  various  phases  of  system 
acquisition.  Certain  definitional  problems  exist  (e.g.,  false 
alarms)  that  should  be  addressed.  One  hundred  percent  diagnos¬ 
tics  are  essential  for  a  successful  system.  A  "thread"  tracing 
CNDs  and  RTOKs  into  the  field  via  a  closed  loop  data  collection 
system  may  be  required  in  order  to  determine  the  causes  for 
system  deficiencies. 

The  sequence  of  issues  is  shown  in  Figure  15-2.  The  final 
issue  is  that  of  communicating  user  needs  to  the  system  devel¬ 
oper  and  the  system  contractor.  The  consensus  was  that  the 
using  command  must  be  in  at  the  beginning  of  the  process  pre¬ 
senting  a  "wish  list"  of  requirements  to  the  system  developer 
that  may  or  may  not  be  achievable  in  terms  of  such  require¬ 
ments  as  turnaround  time,  sortie  generation  and  other  numbers 
of  that  nature.  The  process  of  communicating  these  require¬ 
ments  from  the  using  command  to  the  system  developer  may  be 
assisted  by  a  system  evaluator  (e.g.,  Army,  SEG:  Air  Force, 
AFTEC).  This  type  of  system  evaluator  deals  with  fault  detec¬ 
tion  and  fault  isolation  problems  as  related  to  user  require¬ 
ments  such  as  turnaround  time.  As  such,  the  system  evaluator 
would  act  as  an  interpreter  between  the  "wish  list"  of  the 
user  and  the  procurer's  specification  requirements.  A  great 
deal  of  interaction  between  user  groups,  system  developers, 
and  system  contractors  is  essential  to  this  process.  Realistic 
and  stringent  contractual  specifications  in  terms  of  MTTR, 


‘'SlAX*  reliability  and  availability  requirements  are  needed  to 
establish  the  basic  reliability  and  maintainability  character¬ 
istics  of  the  system,  within  the  constraints  of  the  support 
requirements . 

Support  requirements  demand  a  100  percent  diagnostic  capa¬ 
bility  (automatic  and/or  manual)  to  be  achieved  with  specified 
maintenance  skill  levels.  Nonaddressable  failures  cannot  be 
permitted  since  they  drive  MTTR  excessively  high.  Systems  that 
specify  percentages  of  automatic  FD/FI  (e.g.,  90  percent)  to 
the  contractor  result  in  his  concentrating  on  this  90  percent 
and  neglecting  the  other  10  percent  to  the  detriment  of  overall 
maintainability  (e .g. ,' support ,  training,  etc.).  In  some 
cases,  only  80  percent  automatic  PD/FI  is  achieved;  when  this 
occurs,  the  10  percent  shortcomings  (between  80  and  90  percent) 
has  not  been  planned  for  support  equipment.  The  turnaround 
(when  the  automated  BIT  did  not  work)  presents  extremely  diffi¬ 
cult  maintenance  problems. 

Attempts  to  translate  requirements  such  as  maintenance 
personnel  skill  levels  to  system  maintainability  requirements 
thus  far  have  proved  to  be  relatively  unsuccessful.  It  does 
not  appear  that  industry  has  the  expertise  to  perform  such  a 
translation.  This  may  be  a  worthwhile  area  for  investigation. 

Limitations  should  be  placed  on  the  quantities  of  spares 
available  and  the  number  of  removals  permitted  in  order  to 
prevent  indiscriminate  replacing  of  boxes  to  meet  specified 
MTTR  requirements. 

Limits  on  CNDs  and  RTCKs  should  be  considered  as  possible 
specification  characteristics  to  be  measured  under  certain 
specified  and  controlled  conditions. 

Contractual  incentives  should  be  considered  such  as  Main¬ 
tainability  Improvement  Warranties  (MIW)  comparable  to  RIWs, 
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using  thresholds  and  goals  as  well  as  maintainability  improve¬ 
ment  growth. 

Figure  15-3  indicates  the  requirements  of  system  contrac¬ 
tor  management  including  converting  general  requirements  into 
specific  requirements,  performing  tradeoffs,  and  subcontractor 
management.  This  approach  gives  the  contractor  the  flexibility 
required  to  optimize  the  diagnostic  function.  The  "hard" 
requirement  should  be  MTTR  with  the  contractor  determining  the 
FD  and  FI  percentages  and  the  automatic/manual  breakdown.  In 
all  cases,  the  overall  diagnostic  system  must  provide  100  per¬ 
cent  FD  and  100  percent  FI. 

In  addition  to  the  characteristics  shown,  "Vertical  Testa¬ 
bility"  (capability  of  0,  I,  and  D-level  testing)  is  a  system 
requirement . 

Subcontractor  management  does  not  appear  to  be  a  diffi¬ 
cult  problem,  provided  the  listed  tasks  are  performed.  High 
level  summaries  may  consist  of  data  such  as  percentages  of 
various  functions  covered  by  BIT,  as  opposed  to  analyses  such 
as  FMEAs . 

Figure  15-^  indicates  items  required  during  system  con¬ 
tractual  validation.  Continued  monitoring  and  evaluation  from 
the  beginning  are  required  to  assess  system  maintainability. 

The  possibility  of  making  the  maintainability  demonstra¬ 
tion  of  MIL-STD-(I71  more  realistic  by  combining  it  with  some 
environmental  tests  should  be  considered. 

DISCUSSION  POINTS 

•  The  "smart  box,"  "dumb  man"  concept  apparently  offered 
by  the  extensive  use  of  BIT  is  not  a  viable  concept 
since  highly  skilled  maintenance  personnel  are  re¬ 
quired  ror  the  "beyond  BIT"  maintenance  problems. 
Technicians  capable  of  independent  diagnosis  of  the 
system  faults  are  still  required,  regardless  of  the 
sophistication  of  the  BIT. 
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•  The  number  of  highly  skilled  technicians  versus  the 
number  of  low  skill  level  technicians  is  a  problem  to 
be  addressed  relative  to  the  BIT  capabilities  of  the 
system.  It  has  been  mistakenly  assumed  in  many  cases 
that  no  highly  skilled  technicians  ("smart  guy")  are 
required  because  of  the  capabilities  provided  by  the 
BIT. 

•  "Management"  of  a  program  seems  to  end  at  the  end  of 
development  and  no  suitable  vehicle  presently  exists 
to  adjust  changed  system  maintainability  character¬ 
istics  to  the  production  system. 

•  As  BIT  gets  better  and  better,  the  personnel  skill 
level  problem  is  aggravated  because  the  requirement 
for  a  skilled  maintenance  man  still  exists  and  because 
it  is  easier  to  manage  a  maintenance  staff  with  a  6  to 
1  unskilled  to  skilled  ratio  than  it  is  a  100  to  1 
ratio  staff.  In  addition,  it  is  more  difficult  to 
maintain  high  skill  levels  because  fewer  problems  are 
seen . 

•  Developing  the  training  for  the  second  level  of  support 
(comparable  to  I-level)  in  the  commercial  industry  is 
more  difficult  than  for  the  first  level  (comparable  to 
0-level);  therefore,  it  is  done  first. 

•  The  CND  rate  is  approximately  30  percent  in  the  mili¬ 
tary,  in  industry,  and  the  airlines,  whether  fault 
detection  is  done  automatically  or  manually. 

•  BIT  has  to  be  "tailored"  to  the  type  of  system  in 
which  it  is  incorporated  to  adapt  it  to  the  specific 
constraints  on  the  system. 

•  MTTR  should  be  considered  for  being  specified  as  two 
values,  one  in  the  manual  and  one  in  the  automatic 
mode,  identifying  the  percent  of  time  it  is  performed 
in  each  mode.  Constituents  of  the  MTTR  should  be  con¬ 
sidered  for  specifications  to  address  the  problem  of 
high-time  contributors  to  tota'  MTTR. 

•  MTTR  requirements  must  be  relc.  to  support  costs  so 
that  reductions  in  MTTR  do  not  . esult  in  excessive 
support  costs. 

•  BIT  must  be  used  in  the  way  it  was  planned  for  use  in 
order  to  realize  its  full  benefits. 

•  In  view  of  the  fact  that  two  to  three  years  are  re¬ 
quired  in  operational  use  to  mature  the  BIT,  it  is  not 
possible  to  provide  hard  numeric  values  early  in  the 
program  to  determine  that  contract  requirements  and 
user  constraints  have  been  met.  This  makes  it  diffi¬ 
cult  to  meet  DSARC  requirements  for  such  values. 
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•  Funding  and  personnel  are  required  to  identify  BIT 
problems  and  to  implement  solutions  during  the  two  to 
three  year  BIT  maturing  period. 

•  More  BIT  cleanup  could  be  performed  during  the  system 
development  phase  if  more  attention  were  given  to  this 
aspect  of  the  system,  including  participation  by  users. 
This  requires  early  management  and  design  attention 
applied  to  the  problem. 

•  BIT  design  flexibility  permitting  changes  to  be  imple¬ 
mented  by  software  can  decrease  the  amount  of  time 
required  to  mature  the  BIT,  possibly  permitting  a  one- 
year  maturing  period  instead  of  the  current  two  to 
three  year  period. 
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ISSUES/CBSERVATIONS 


1.  BIT  FIELD  PERFORMANCE  LESS  THAN  SPECIFIED  AND 
DEMONSTRATED, 

2.  CUM  APPROACHES  DO  NOT  TAKE  INTO  ACCOUNT  FALSE 
ALARMS  UNDETECTED  FAILURE,  OK,  CND, 

3.  BIT  FERFOemNCE  NOT  COMPATIBLE  WITH  MANPOCR  AND 
LOGISTIC  REQUIREMENTS. 

A,  INADEQUATE  RESOURCE  AND  SCHEDULE  ALLOCATION. 


Figure  15-1 


SMTlflMPACM  SPECIFICATIONS 
SffiMNT  BEflJLJ m  SPtCIFICATlONS 

1.  FGHmKKE  TEQUIFBtNT 

-  fTTTfL  fH\x 

-  MTBF 

-  AVAILABILITY 

2.  SUPPORT  FEQUiHJBFS 

-  m  DIAGNOSTIC  CAPABILITY 

-  MAINTENANCE  SKILL  LEVELS 

-  SPATES  LIMITATIONS  (MTEW 

-  CND,  FOCK  LIMITS 

3.  CONTRACTUAL  INCENTIVES 

-  WARRANTIES 

-  THRESHOLD:  GCALS 


Figure  15-3 


p  :  I 

CONVERT  GENERAL  REQUIRS’ENTS  INFO  SPECIFIC  REQUIREMENTS 

-  FD/FI 

-  AUTOMATIC 

-  MANUAL 

PERFORM  TRADEOFFS 

-  BIT  ARCHITECTURE 

-  BIT  TECHNOLOGY 

-  OF  THE  SHELF  VS,  LEW  DESIGN 

-  MAINTENANCE  VS.  OPERATOR  BIT 

-  INCORPORATE  BIT  DATA  INTO  I-LEVEL  TESTING 


SUBCONTRACTOR  MANAGEMENT 

-  ALLOCATE  SPECIFIC  REQUIREMENTS 

-  COMMUNICATE  SYSTEM  DESIGN  APPROACH 

-  DATA  REQUIREMENT 


SYSTEM  CONTRACTOR 

TEST  STRUCTURED  FMEA 
BIT  ACCURACY 
TOLERANCE  LEVELS 
TEST  POINT  SELECTION 


SYSTEM  DEVELOPER 

•  HIGH  LEVEL  SUMMARY  -  REGULAR  BASIS 

•  DETAILED  LEVEL  -  AVAILABLE  ON  REQUEST 


SYSTEM  CONTRACTUAL  VALIDATION 
Figure  15-A 

EEVEUDPMENT  OF  TEST  FACILITIES  &  HARDWARE 
CONTINUAL  EVALUATION  &  MIlESTCfES  (TAAF) 
FEALISTIC  M-EEH) 

CCfBIie  ENVIROfMFNTAL  TESTING 
SIMULATE  OPERATIONAL  ENVIROfMENT 


WPOt£R 

T.O.s 


-  OPERATIONAL  EVALUATION 


SYSTEM  SHAKEDOWN 
FALSE  ALARMS,  CND5 

UNANTICIPATED  OPERATIONAL  ENVIRONMENTS 
UNANTICIPATED  FAILURE  WEES 

EVALUATE  PERFORMWTCE  REQUIREMENTS 
EVALUATE  SUPPORT  RETIREMENTS 
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Berlin,  Howard 
US AMS A A 

Aberdeen  Proving  Ground,  MD  21005 
Attn:  DRXSY-PR 

Boardman,  Howard 
Bldg  127-307 

RCA,  Missiles  and  Surface  Radar  Division 
Moorestown,  NJ  08057 

Bragg,  Lucas 

Logistics  Management  Institute 
4701  Sangmore  Road 
Washington,  DC  20016 

Charles,  Rick 
ARINC,  Avionics  Division 
2551  Riva  Road 
Annapolis,  MD  21404 

Chen,  Wei  Long 
Control  Data  Corooraticn 
3101  E.  80th  Street,  HQG  319 
Bloomington,  MN  55440 

Coppola,  Tony 
RADC/RBET 

Griffis  APB ,  NY  13441 

Danforth,  LTC  Bill 
USA  OTEA 

5600  Columbia  Pike 
Annandale,  VA  22041 

Drummond ,  Bob 

McDunnell  Aircraft  Company 

PO  Box  516 

St.  Louis,  MO  63166 

Dudek,  Stan 
IBM-FSD 

10215  Fernwood  Road 
Bethesda,  MD  20034 

England,  Gordon 
General  Dynamics 
PO  Box  748,  Fort  Worth  Division 
Fort  Worth,  TX  76101 


Faggard,  Major  Gene 
HQ  AFSC/SDDP 
Andrews  AFB,  MD  20331 
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Genet,  Russ 
AFHRL/LRLA 

Wright  Patterson  AFB,  OH  ^5^33 

Gleason,  Caotain  Dan 
RADC/RBET 

Griffiss  AFB,  NY  13441 

Giiffith,  Homer 
Project  Manager 
PARTIOT  Missile  System 
Redstone  Arsenal,  AL  35901 
Attn:  CRCPM-MI-T-C 

Gunkel,  Colonel  R. 

HQ  AFTEC/LG 

Kirtland  AFB,  NM  87117 

*Hardison,  The  Honorable  David  D. 
Deputy  Under  Secretary  of  Defense  R&E 
Tactical  Warfare  Programs 
Room  2E1044,  Pentagon 
Washington,  DC  20301 

Jenkins,  Bob 
NAVSEA  SYSCOM  HQ 
PM5-508 

Washington,  DC  20362 

Kauffman,  Bob 
US  Army  CORADCOM 
DRDCO-CTL— M 

Fort  Monmouth,  NJ  07703 

Keiner,  Bill 
Code  F105 

Naval  Surface  Weaoons  Center 
Dahlgren,  VA  224^8 

*Klett,  Capt  G.J. 

Naval  Materiel  Command 
348  CP5 

2211  Jefferson  Davis  Highway 
Arlington,  VA  20360 

Linden,  Major  Vince 
HQ  AFTEC/LGL 
Kirtland  AFB,  NM  87117 


*Senior  Board  Member 


A-2 


*Lorber,  Seymour 

Army  Materiel  Development  &  Readiness 
Command  Headquarters 
4W22  AMC 

5001  Eisenhower  Avenue 
Alexandria,  VA  22333 

McCarthy,  Bill 
Raytheon  Corporation 
Hartwell  Road 
Bedford,  MA  07130 

McCough,  B.J. 

Wescinghouse  Defense  &  Electronics  Center 
P0  Box  7 46,  MS  1620 
Baltimore,  MD  21203 

McGrath,  Mike 

Office  of  the  Secretary  of  Defense 
OASD  (MRA&L)  WS 
Room  2B322,  Pentagon 
Washington,  DC  20301 

Meth,  Martin 

Office  of  the  Secretary  of  Defense 
0ASD( MRA&L)  WS 
Room  2B322,  Pentagon 
Washington,  DC  20301 

Meyer,  Ed 

McDonnell  Douglas  Corporation 
PO  Box  516,  Dept  311,  Bldg  1 
St.  Louis,  MO  63166 

Mileson,  Don 
P0  Drawer  K 
McLean,  VA  22301 

Mussor.,  Colonel  Tom 
Assistant  for  R&M 
OUSD  ( R&E)  SS 
Room  2A318,  Pentagon 
Washington,  DC  20301 

Neumann,  George 
NAVMAT-O^T 

Washington,  DC  20301 

Nunn,  Mel 
Code  921 

Naval  Ocean  Systems  Center 
San  Diego,  CA  92152 

*Senior  Board  Member 
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O'Brien,  Mike 

Institute  for  Defense  Analyses 
400  Army-Navy  Drive 
Arlington,  VA  22202 

Petersen,  Norm 

Westinghouse  Defense  &  Electronics  Center 
PO  Box  746,  MS  1620 
Baltimore,  MD  21203 

Pontlous,  Gary 

General  Dynamics,  Pomona  Division 
PO  Box  2507,  MZ  28-6 
Pomona,  CA  91766 

Pruitt,  Kelly 
Project  Manager 
PATRIOT  Missile  System 
Redstone  Arsenal,  AL  35901 
Attn:  DRCPM-MD-S-P 

Rogers,  Jim 

Staff  Assistant,  DDT&E 
OSD/OUSDR&E/DDT&E 
Room  3D1043,  Pentagon 
Washington,  DC  20301 

Rogers,  John 
Naval  Avionics  Center 
600  E.  21st  Street 
Indianapolis,  IN  46218 

Schumacher,  Lee 
DoD  PESO 

c/o  DLA,  Cameron  Station 
Alexandria,  VA  223i4 

Shafer,  Major  Bob 
AFTEC  OL-AD/AF 
Hill  AFB ,  UT  84506 

*Shorey,  Russell  R. 

Special  Assistant  for  Weapons  Support 
0ASD(MRA&L) 

Room  2B322,  Pentagon 
Washington,  DC  20301 
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^Senior  Board  Member 


Vershish,  LCDR  Bob 
COMOPTEVPOR 
So.  Division 
Naval  Base 
Norfolk,  VA  23511 

Victor,  Jim 

Westinghouse  Defense  &  Electronics  Center 
PO  Box  746,  MS  1620 
Baltimore,  MD  21203 

Wares,  Ed 
NAVSEA  SYSCOM  HQ 
AEGIS  PMS 

V/ashington,  DC  20362 
*Watt,  Charles 

Deputy  for  Strategic  and  Naval  Warfare  Systems 
OUSD  (R&E) 

Room  2B284,  Pentagon 
Washington,  DC  20301 

Wes s el,  LTC  J. 

HQAFTEC/LGL 

Kirtland  AFB ,  NM  87117 

Wheeler,  Raymond 
AFTEC  OL-AD/FM 
Hill  AF3 ,  UT  84506 

Williams,  Harry 

Institute  for  Defense  Analyses 
400  Army-Navy  Drive 
Arlington,  VA  22202 

Wilson,  Ken 

9819  Brentsville  Road 

Manassas,  VA  22110 

^Winters,  Colonel  George 
HQ  Air  Force  Systems  Command 
HQ  AFSC/SDTA 
Andrews  AFB,  MD  20334 

Wood,  Bob 
BLDG  116-302 

RCA  Missile  and  Surface  Radar  Division 
Moorestown,  NJ  08057 
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Zabeli,  Arthur 
ARINC,  Avionics  Department 
2551  Riva  Road 
Annapolis,  MD  21^01 
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